Musical performance assistance method, musical performance assistance system, and program

Abstract

A musical performance assistance method implemented by a computer according to an embodiment comprises: a performance data acquisition step for acquiring performance data pertaining to a performance by a user; a part acquisition step for acquiring first part data corresponding to a predetermined first part from musical piece data including a plurality of parts; an identification step for identifying a performance position of the performance by the user on the basis of the correspondence relationship between the performance data and the musical piece data; and a playback step for playing the first part data according to the identified performance position.

Description

Performance support method, performance support system and program

[0001] The present invention relates to a performance assistance method, a performance assistance system, and a program.

[0002] There are known techniques for extracting data corresponding to a specific part from data for multiple parts, including a vocal part. For example, Patent Document 1 discloses a technique for extracting score data for a specific part from score data for multiple parts (full score data). Also, Patent Document 2 discloses a technique for generating a score image for each part from score images for multiple parts (full score images), and at that time, reflecting instruction symbols in the full score in the score image for each part.

[0003] JP 2003-288075 A JP 2021-152618 A

[0004] One of the objects of the present invention is to give a user a sense of unity as if other people were playing along with the user's performance.

[0005] In one embodiment, a performance assistance method implemented by a computer includes a performance data acquisition step of acquiring performance data of a performance by a user, a part acquisition step of acquiring first part data corresponding to a predetermined first part from song data including multiple parts, an identification step of identifying a performance position of the performance by the user based on the correspondence between the performance data and the song data, and a playback step of playing back the first part data in accordance with the identified performance position.

[0006] According to the present invention, it is possible to give a user a sense of unity as if other people are playing along with the user's performance.

[0007] 1 is a block diagram showing a performance assistance system according to one embodiment of the present invention; FIG. 2 is a block diagram showing an example of the configuration of a terminal; FIG. 3 is a block diagram showing an example of the configuration of a performance device; FIG. 4 is a block diagram showing the configuration of a performance assistance function according to this embodiment; FIG. 5 is an example of a screen on which an image showing chord information in a visually recognizable manner is displayed; FIG. 6 is an example of a screen on which an image showing chord information in a visually recognizable manner is displayed; FIG. 7 is a flowchart for explaining an example of performance assistance processing executed by the performance assistance function according to this embodiment; FIG. 8 is a block diagram showing an example of the configuration of the performance assistance function of this modified example;

[0008] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. The embodiment shown below is merely an example, and the present invention should not be construed as being limited to these embodiments. In the drawings referred to in this embodiment, identical parts or parts having similar functions are designated by the same or similar reference numerals, and repeated description thereof may be omitted.

[0009] [Configuration of the Performance Assistance System] Fig. 1 is a block diagram showing a performance assistance system according to one embodiment of the present invention. The performance assistance system 1 includes a terminal 10 and a performance device 20. The terminal 10 and the performance device 20 can communicate with each other via a wired or wireless connection. For example, the terminal 10 and the performance device 20 may be connected via a wired connection such as a USB connection, or via short-range wireless communication such as Bluetooth (registered trademark).

[0010] The terminal 10 is a device equipped with a computer, such as a desktop terminal, a laptop terminal, a smartphone, a tablet terminal, etc. The terminal 10 provides the user with a performance assistance function by executing an installed application program.

[0011] The performance device 20 is an electronic musical instrument that has performance controls and outputs sounds based on the performance operations of the user. In this embodiment, the performance device 20 is an electronic piano, as an example.

[0012] [Configuration of Terminal] The configuration of the terminal 10 will be described. Fig. 2 is a block diagram showing an example of the configuration of the terminal 10. The terminal 10 includes a control unit 11, a storage unit 12, a communication unit 13, an operation unit 14, a display unit 15, an interface 16, and a speaker 17. The control unit 11, the storage unit 12, the communication unit 13, the operation unit 14, the display unit 15, the interface 16, and the speaker 17 are connected to one another via a bus 18.

[0013] The control unit 11 includes an arithmetic processing circuit such as a CPU (Central Processing Unit), and storage devices such as a RAM (Random Access Memory) and a ROM (Read Only Memory). The control unit 11 executes a program 12a stored in the storage unit 12 using the CPU to realize various functions in the terminal 10. The functions executed by the control unit 11 include a performance assistance function. The performance assistance function executed by the control unit 11 will be described later.

[0014] The storage unit 12 is a storage device such as a non-volatile memory. The storage unit 12 stores a program 12a that is executed by the control unit 11 and that realizes the performance assistance function. The program 12a may be provided in a state stored in a computer-readable recording medium such as a magnetic recording medium, an optical recording medium, a magneto-optical recording medium, or a semiconductor memory, as long as it is executable by a computer. In this case, the terminal 10 may be provided with a device for reading the recording medium. The program 12a may also be downloaded via a network.

[0015] The storage unit 12 also stores information necessary for the control unit 11 to execute the performance assistance function, including song data 12b and trained model 12c. Such information, including song data 12b and trained model 12c, may be stored in an external storage device (server, recording medium, etc.) connectable via wire or wirelessly, instead of being stored in the storage unit 12. The storage unit 12 may also include a storage area for temporarily storing data output from the trained model 12c, etc.

[0016] The song data 12b is sound waveform data including multiple parts of sound corresponding to a predetermined song. The song data 12b may be, for example, data in WAV (Waveform Audio File Format). In other words, the song data 12b is a single piece of sound waveform data including a vocal part. The song data 12b may be downloaded via a network. The storage unit 12 stores song data 12b corresponding to one or more songs.

[0017] The trained model 12c is obtained by machine learning. The trained model 12c is a computational model used to identify a user's performance position in a predetermined piece of music from input performance data. Here, the performance data input to the trained model 12c is sound generation control data such as MIDI (Musical Instruments Digital Interface) generated based on performance operations performed by the user when playing the predetermined piece of music. A known machine learning model is applied to the trained model 12c. The known machine learning model is, for example, a model using a neural network that utilizes a CNN (Convolutional Neural Network), an RNN (Recurrent Neural Network), or the like.

[0018] The trained model 12c is a trained model obtained by machine learning the correlation between the performance data and the song data 12b. Specifically, the trained model 12c is a trained model obtained by machine learning the correlation between the performance data and the chord progression and performance position of a predetermined song, and is a trained model obtained by learning the performance content when various performers play the predetermined song. When chord information of the predetermined song and performance data based on performance operations when a user plays the predetermined song are input, the trained model 12c outputs information indicating the user's performance position in the predetermined song (hereinafter referred to as performance position information) corresponding to the input data. The chord information and performance data will be described later.

[0019] The communication unit 13 is a communication module that connects to a network under the control of the control unit 11 and communicates with external devices connected to the network.

[0020] The operation unit 14 is an input device such as an operation button, a keyboard, a mouse, etc., provided on an operation panel or a remote control, and outputs a signal according to an input operation to the control unit 11. The operation unit 14 enables the user to input instructions to the terminal 10.

[0021] The display unit 15 is a display device such as a liquid crystal display or an organic EL display, and displays a screen based on the control by the control unit 11. The operation unit 14 and the display unit 15 may be integrated to form a touch panel.

[0022] The interface 16 includes a module for communicating with the performance device 20, external speakers, etc. via wired communication or wireless communication such as infrared communication or short-range wireless communication. The interface 16 also includes a USB interface such as a USB port, an interface for Bluetooth (registered trademark) connection, etc.

[0023] The speaker 17 converts sound data output from the control unit 11 into air vibrations and provides the sound to the user.

[0024] [Configuration of Performance Device] The configuration of performance device 20 will be described. Fig. 3 is a block diagram showing an example of the configuration of performance device 20. Performance device 20 includes a control unit 21, a storage unit 22, a sound source unit 23, performance operators 24, a sensor 25, an interface 26, and a speaker 27. Control unit 21, storage unit 22, sound source unit 23, sensor 25, interface 26, and speaker 27 are connected to one another via a bus 28.

[0025] The control unit 21 includes an arithmetic processing circuit such as a CPU, and storage devices such as a RAM and a ROM. The control unit 21 executes a program 22a stored in the storage unit 22 using the CPU to realize various functions in the performance device 20. The functions executed by the control unit 21 include a signal conversion function.

[0026] The storage unit 22 is a storage device such as a non-volatile memory. The storage unit 22 stores a program 22a that is executed by the control unit 21 and that implements various functions in the performance device 20. The program 22a may be provided in a state stored in a computer-readable recording medium such as a magnetic recording medium, an optical recording medium, a magneto-optical recording medium, or a semiconductor memory, as long as it is executable by a computer. In this case, the performance device 20 may be provided with a device for reading the recording medium. The program 22a may also be downloaded via a network.

[0027] The sound source unit 23 includes a DSP (Digital Signal Processor). When a performance is performed on the performance device 20 based on the operation of the performance operators 24, the sound source unit 23 generates sound data, which is sound waveform data, based on performance data based on the performance operation. The performance data will be described later.

[0028] The performance operators 24 are operators that are operated by the user for performance purposes. The performance operators 24 include a plurality of keys and pedals. A sensor 25 is provided corresponding to each of the performance operators 24.

[0029] The sensor 25 detects the operation of the performance operators 24 and outputs a detection signal according to the detected content. For example, the sensor 25 outputs a detection signal according to the amount of depression of each key or pedal.

[0030] The control unit 21 generates performance data corresponding to the operation state of the performance operators 24 based on the detection signals output from the sensor 25. The performance data is sound generation control data that specifies sound generation, and is, for example, data in MIDI format. The performance data generated in response to key operation includes note-on, note-off, note number, and velocity indicating the key pressing speed. In the performance data generated in response to key pressing, the note number and velocity are output in correspondence with note-on. In addition, in the performance data generated in response to key release, the note number is output in correspondence with note-off.

[0031] The interface 26 includes a module for communicating with the terminal 10, an external speaker, etc., via wired communication or wireless communication such as infrared communication or short-range wireless communication. The interface 26 includes a USB interface such as a USB port, Bluetooth (registered trademark), etc.

[0032] The speaker 27 converts the sound data output from the sound source unit 23 into air vibrations and provides the sound to the user.

[0033] [Software Configuration of Terminal] The following describes a performance assistance function that is realized by the control unit 11 of the terminal 10 executing the program 12a. The terminal 10 extracts data corresponding to the vocal part from the song data corresponding to the song designated by the user, and plays back the extracted data corresponding to the vocal part in time with the user's performance on the performance device 20.

[0034] 4 is a block diagram showing the configuration of the performance assistance function 100 according to this embodiment. Part or all of the configuration for realizing the performance assistance function 100 described below may be realized by hardware. The performance assistance process executed by the performance assistance function 100 is started by an instruction from the user via the operation unit 14.

[0035] Referring to FIG. 4, the performance assistance function 100 includes a generation unit 101 , a chord analysis unit 102 , a performance data acquisition unit 103 , a performance position identification unit 104 , a provision unit 105 , a playback unit 106 , and a determination unit 107 .

[0036] Predetermined song data 12b stored in the storage unit 12 is input to the generation unit 101 and the chord analysis unit 102. The predetermined song data 12b input to the generation unit 101 and the chord analysis unit 102 is song data 12b corresponding to a song designated by the user via the operation unit 14. When the user designates a song, the name of the song and the artist name corresponding to the song data 12b stored in the storage unit 12 of the terminal 10 may be displayed on the display unit 15. The user can select the song he or she wants to play based on the displayed song name and artist name.

[0037] The generation unit 101 (part acquisition unit) generates (acquires) first part data corresponding to the vocal part from the song data 12b. As described above, the song data 12b includes a vocal part. The generation unit 101 extracts sound waveform data corresponding to the vocal part of the musical piece from the song data 12b, separates it from the song data 12b, and generates the first part data. Note that the extraction of sound waveform data corresponding to the vocal part from the song data 12b can be performed using various known audio editing techniques. The generation unit 101 may also automatically extract sound waveform data corresponding to the vocal part from the song data 12b using AI (Artificial Intelligence). The first part data is associated with time information that specifies playback timing. The generation unit 101 outputs the first part data corresponding to the vocal part to the playback unit 106.

[0038] The chord analysis unit 102 analyzes the song data 12b to generate (obtain) chord information corresponding to the song data 12b. The chord information is information indicating the chords of the song corresponding to the song data 12b, and includes information regarding the chord symbol corresponding to the chord, the note on the staff corresponding to the chord, or the key on the keyboard corresponding to the chord. The chord analysis unit 102 analyzes all (or any one or more) performance parts constituting the song, including the vocal part, to generate chord information. The chord information also includes a chord string (information indicating a chord progression) in which information indicating the chords of the song is arranged in chronological order. The chord information is associated with time information that specifies the playback timing. The chord analysis unit 102 outputs the generated chord information to the performance position identification unit 104 and the providing unit 105. Note that, if sound generation control data (e.g., MIDI data) for the song is present, the chord analysis unit 102 may generate chord information by analyzing the sound generation control data.

[0039] The performance data acquisition unit 103 acquires performance data from the performance device 20 via the interface 16. As described above, the performance data is sound generation control data such as MIDI data generated in response to the user's performance operation on the performance device 20. It is assumed that the user plays, on the performance device 20, a piece of music corresponding to the song data 12b input to the generation unit 101 and the chord analysis unit 102. The performance data acquisition unit 103 outputs the acquired performance data to the performance position identification unit 104.

[0040] The performance position identification unit 104 identifies the user's performance position in the song data 12b based on the correspondence between the chord information obtained from the chord analysis unit 102 and the performance data obtained from the performance data acquisition unit 103. The performance data may be performance data corresponding to a predetermined chord, or may be performance data corresponding to a single note that constitutes the predetermined chord.

[0041] The performance position identification unit 104 identifies the user's performance position using the trained model 12c. Therefore, when chord information and performance data are input, the performance position identification unit 104 outputs performance position information indicating the user's performance position (performance location) in the song data 12b corresponding to the song being played by the user. The performance position identification unit 104 outputs the performance position information to the providing unit 105 and the playback unit 106.

[0042] The providing unit 105 acquires code information from the code analysis unit 102 and provides the acquired code information in a visually recognizable (visible) manner. Specifically, the providing unit 105 generates image data for displaying the code information as an image visually recognizable by a user on the display unit 15. The providing unit 105 may provide the image in which the code information is provided in a visually recognizable manner as a GUI (Graphical User Interface).

[0043] 5 and 6 each show an example of a screen on which an image showing visually recognizable code information is displayed. A screen 500 shown in Fig. 6 and a screen 500A shown in Fig. 5 each include a setting area SA and a code information display area CA.

[0044] The setting area SA includes, for example, the name of the song MN, a link URLN to an external site, and operation images GC1 to GC8 for receiving user instructions. The operation image GC1 is an image modeled after a button for receiving instructions such as to start playback and stop playback. The operation image GC2 is an image modeled after a slider for receiving a setting for the playback position in the song being played. The operation image GC2 may also display an indicator of the playback position in the song being played. The operation image GC3 is an image modeled after a slider for receiving a setting for the playback mode of the first part data set by the terminal 10. The operation image GC4 is an image modeled after a slider for receiving a setting for the playback speed of the first part data. The operation image GC5 is an image modeled after a slider for receiving a setting for the playback volume of the first part data. The operation image GC6 is an image modeled after a button for receiving instructions to start and stop recording. The operation image GC7 is an image modeled after a button for receiving an instruction to end performance. Operation image GC8 is an icon for receiving an instruction to display a setting screen (not shown).

[0045] The playback modes that can be set using the operation image GC3 may include, for example, a mode (mode 1) in which the first part data is played back in synchronization with the user's performance on the performance device 20, and a mode (mode 2) in which the first part data is played back without completely matching the user's performance. In mode 1, the playback unit 106 (described later) plays back the first part data in synchronization with the user's performance on the performance device 20 and outputs it to the speaker 17. That is, in mode 1, when the user's performance speed changes, the playback speed of the first part data changes in synchronization with the user's performance. Also, in mode 1, when the user's performance stops, the playback of the first part data stops in synchronization with the user's performance. On the other hand, in mode 2, even if the user's performance operation on the performance device 20 stops, the first part data is played back and output to the speaker 17, but the playback speed is slowed down. Note that in mode 2, the playback speed does not have to be slowed down even if the user's performance operation stops, and the playback speed may always be constant.

[0046] The chord information display area CA displays chord information in a manner that is visually recognizable to the user. The chord information displayed in the chord information display area CA includes a chord string CL in which information indicating the chords of the song data 12b is arranged in chronological order. In this example, the chord string CL displays chord information for each measure of the music corresponding to the song data 12b in chronological order, and indicates the chords in each measure.

[0047] When the user starts a performance operation on the performance device 20, the performance position identification unit 104 provides performance position information to the providing unit 105. Based on the performance position information, the providing unit 105 can provide the user's performance position on the screen of the display unit 15 in a visually recognizable manner.

[0048] On screens 500 and 500A, the user's playing position in the chord sequence CL is displayed in a visually recognizable manner. In this example, the measure PA in which the user is playing is displayed in a different color from the other measures. The user's playing position within a measure may also be displayed in more detail. In this example, the user's playing position within a measure (the measure PA in which the user is playing) is indicated by a bar 501. The measure PA in which the user is playing and the position of the bar 501 change based on playing position information provided by the playing position identification unit 104. By visually displaying the user's playing position and the chord sequence CL arranged in chronological order, the user can recognize the note to be played currently (the constituent notes of the chord) and the note to be played next (the constituent notes of the chord). Note that the number of chord information displayed in each measure is not limited to one. If the chord changes within a measure, as shown in the measure PA, two or more chord information pieces may be displayed.

[0049] The chord information displayed in the chord information display area CA may also include performance chord display information CI indicating the chord CC corresponding to the user's playing position and the next chord NC. The chord CC and chord NC may be displayed as notes on a staff or as keys on a keyboard. On the screen 500 shown in FIG. 6, the chords CC and NC are displayed as notes on a staff, while on the screen 500A shown in FIG. 5, the chords CC and NC are displayed as keys on a keyboard. The display format of the chords CC and NC can be set using an operation image GC9 for setting the chord display format of the performance chord display information CI. The operation image GC9 is an image simulating a slider for accepting the setting of the chord display format.

[0050] In this way, by displaying the chord CC and the next chord NC corresponding to the user's playing position separately from the chord string CL, the user can easily identify the notes that make up the chord corresponding to the playing position or the key position on the keyboard that corresponds to the chord, thereby assisting the user in performing the performance. The operation image GC9 may be displayed in the setting area SA instead of the chord information display area CA. The display of the performance chord display information CI in the chord information display area CA may be switched on and off by the user. The performance chord display information CI may not only display the chord CC and the next chord NC corresponding to the user's playing position, but may also display chords up to a predetermined number of bars or chords after the user's playing position (e.g., a predetermined number of bars or chords after the user's playing position).

[0051] The providing unit 105 generates image data for displaying an image such as that shown on the screen 500 or the screen 500A based on the chord information and the performance position information. The providing unit 105 outputs the generated image data to the display unit 15. The display unit 15 displays an image based on the acquired image data.

[0052] Returning to FIG. 4 , the playback unit 106 acquires the first part data from the generation unit 101 and plays back the first part data in accordance with the identified performance position. For example, the playback unit 106 acquires performance position information from the performance position identification unit 104 and, based on the performance position information, plays back a portion of the first part data corresponding to the performance position (data for playback from the identified performance position) and outputs the data to the speaker 17. Note that this processing by the playback unit 106 can also be expressed as "playing back the first part data corresponding to the identified performance position," "playing back the first part data at the identified performance position," or "playing back the first part data at the identified performance position." Furthermore, this processing by the playback unit 106 can also be expressed as "using the first part data to play back the audio of the first part of the song indicated by the song data 12b (the song being played by the user) from a playback position corresponding to the identified performance position."

[0053] The determination unit 107 determines whether a condition for ending the performance assistance process is satisfied. For example, the determination unit 107 ends the performance assistance process when the user's performance position identified by the performance position identification unit 104 reaches the end position of the song data 12b or when the user inputs an instruction to end the performance. Note that the determination unit 107 may also end the performance assistance process when a predetermined time has elapsed since the user stopped playing.

[0054] In this way, the terminal 10 can play back the first part data generated from the song data 12b in synchronization with the user's playing position on the performance device 20. In other words, the user's playing position coincides with the playback portion of the first part data played back by the playback unit 106 of the terminal 10. As described above, the first part data corresponds to the vocal part of the song played by the user. This allows the user to experience as if the singer were singing along with their own playing.

[0055] In this way, the performance assistance function 100 acquires performance data of a performance by the user and acquires first part data corresponding to a predetermined first part from the song data 12b, which includes multiple parts. For example, the performance assistance function 100 acquires first part data corresponding to a part other than the part performed by the user, such as a vocal part. Then, the performance assistance function 100 identifies a performance position of the user's performance based on the correspondence between the performance data and the song data 12b, and plays back the first part data in accordance with the identified performance position. In other words, the performance assistance function 100 plays back the first part data from the identified performance position. As a result of this processing, the performance assistance function 100 can give the user a sense of unity, as if someone else were playing along with the user's performance.

[0056] The performance assistance function 100 also acquires chord information corresponding to the song data 12b. The performance assistance function 100 then identifies the user's performance position based on the correspondence between the chord information and the performance data. In this way, the performance assistance function 100 identifies the user's performance position using the chord information, so the user's performance position can be identified with high accuracy.

[0057] The performance assistance function 100 also provides visually recognizable chord information generated from the song data 12b. For example, the performance assistance function 100 generates information including a chord string in which information indicating chords is arranged in chronological order as chord information. The performance assistance function 100 then provides a visually recognizable chord corresponding to the performance position. The performance assistance function 100 may also provide a visually recognizable chord following the chord corresponding to the performance position. In this way, the performance assistance function 100 provides visually recognizable chord information, making it easier for the user to recognize which part is being played.

[0058] Furthermore, the performance assistance function 100 identifies the performance position of the user's performance using a trained model 12c that has been trained to learn the correlation between the performance data and the song data 12b. For example, the performance assistance function 100 identifies the performance position of the user's performance using the trained model 12c that has been trained to learn the correlation between the performance data of a predetermined song and the chord progression in the song data 12b and the performance position in the performance data. In this way, the performance assistance function 100 identifies the user's performance position using a trained model that identifies the user's performance position based on the chord progression, thereby enabling the performance position to be identified with high accuracy.

[0059] The performance assistance function 100 also acquires the first part data from the song data 12b, which is sound waveform data. Therefore, the performance assistance function 100 can use sound waveform data such as WAV to reproduce the first part data in accordance with the user's performance.

[0060] 7 is a flowchart illustrating an example of the performance assistance process executed by the performance assistance function 100 according to this embodiment. The flowchart of the performance assistance process shown in FIG. 7 is merely an example, and the process may be performed in any order. For example, steps S301 and S302 may be performed in reverse order, or may be performed simultaneously in parallel. The performance assistance process is initiated by a user's instruction via the operation unit 14.

[0061] The generation unit 101 generates first part data corresponding to a vocal part from the song data 12b corresponding to a song designated by the user (S301), and the chord analysis unit 102 generates chord information corresponding to the song data 12b (S302).

[0062] The providing unit 105 provides the chord information in a visually recognizable manner (S303). Here, providing the chord information in a visually recognizable manner includes generating image data for displaying the chord information on the display unit 15 as an image that is visually recognizable to the user, and displaying an image based on the image data on the display unit 15. The visually recognizable chord information includes a chord string in which information indicating the chords of the song data 12b is arranged in chronological order.

[0063] When the user starts a performance operation on the performance device 20, the performance data acquisition unit 103 acquires performance data corresponding to the performance operation (S304). The performance position identification unit 104 identifies the user's performance position in the song data 12b based on the correspondence between the chord information in the song data 12b and the performance data (S305).

[0064] The playback unit 106 plays the first part data in accordance with the identified playing position (S306). In other words, the performance assistance function 100 identifies a playing position of the first part data based on the playing position, and plays the first part data from the identified playing position. Although not shown, the played first part data is provided to the speaker 17 and output from the speaker 17. In addition, the providing unit 105 provides the chords of each part at the playing position identified by the playing position identifying unit 104, and the chords for a predetermined number of parts after the identified playing position (S307).

[0065] Next, the determination unit 107 determines whether a termination condition for the performance assistance process is satisfied (S308). The termination condition for the performance assistance process is, for example, when the user inputs an instruction to end the performance, or when the user's performance position reaches the end position of the song data 12b. If the termination condition is satisfied (S308: Yes), the determination unit 107 terminates the performance assistance process. On the other hand, if the termination condition is not satisfied (S308: No), the performance data acquisition unit 103, the performance position identification unit 104, the provision unit 105, the playback unit 106, and the determination unit 107 repeatedly execute the processes of S304 to S307.

[0066] [Modifications] The present invention is not limited to the above-described embodiment, and includes various other modifications, some of which will be described below.

[0067] (1) In the above embodiment, the performance assistance function 100 analyzes all parts included in the musical piece corresponding to the song data 12b and generates chord information corresponding to the song data 12b. However, the performance assistance function 100 may generate chord information for any part. For example, the performance assistance function 100 may first obtain first part data corresponding to a vocal part from the song data 12b, and then generate chord information from the song data 12b after the first part data has been extracted and separated (i.e., song data 12b that does not include a vocal part). In other words, the performance assistance function 100 may generate chord information from data for a part other than the part corresponding to the first part data, as long as it generates chord information from data for at least some of the parts included in the musical piece.

[0068] The performance assistance function 100 may also play any part in time with the user's performance. For example, the performance assistance function 100 may receive from the user a selection of a part to be played in time with the user's performance from among multiple parts included in a musical piece. The performance assistance function 100 may then obtain first part data corresponding to the part selected by the user and play the obtained first part data in time with the user's performance.

[0069] The performance assistance function 100 may also play any number of parts in accordance with the user's performance. For example, the performance assistance function 100 may acquire first part data corresponding to a first part and second part data corresponding to a second part different from the first part from the song data 12b, and play the first and second part data in accordance with the user's performance. More specifically, the performance assistance function 100 may acquire first part data corresponding to a vocal part including vocals and second part data corresponding to a drum part including drums from the song data 12b. The performance assistance function 100 may then play the first and second part data in accordance with the user's performance. The performance assistance function 100 may also acquire part data corresponding to three or more parts from the song data 12b and play the acquired part data in accordance with the user's performance. As a result, the performance assistance function 100 can give the user a sense of unity as if multiple parts were being played by others in accordance with the user's performance (more specifically, a sense of unity as if a band were playing in accordance with the user's performance).

[0070] Here, the performance assistance function 100 may generate first part data corresponding to the same part as the part played by the user and play the generated first part data. In this way, when the first part data of the same part as the part played by the user is played, the performance assistance function 100 can realize performance assistance, for example, by allowing a user who is not confident in their performance to play a certain part while relying on the sound played by the performance assistance function 100.

[0071] Furthermore, the performance assistance function 100 may generate first part data corresponding to a part different from the part played by the user and play the generated first part data. In this way, when the first part data of a part different from the part played by the user is played, the performance assistance function 100 can, for example, follow the user's performance when the user plays a certain part, providing the user with a sense of unity as if other parts were being played by others. When playing multiple part data corresponding to multiple parts, the performance assistance function 100 may play back first part data corresponding to the same part as the part played by the user, or may play back only first part data corresponding to a part different from the part played by the user.

[0072] (2) In the above embodiment, sound waveform data corresponding to the vocal part is extracted and separated from the song data 12b. However, the generation unit 101 can extract and separate sound waveform data corresponding to the sound of at least one part specified by the user, included in the song data 12b, in addition to or instead of the sound waveform data corresponding to the vocal part. In this case, the generation unit 101 extracts and separates sound waveform data corresponding to the sound of at least one part specified by the user from the song data 12b and generates it as second part data. The second part data is played back in accordance with the specified performance position, just like the first part data.

[0073] Furthermore, when reproducing the first and second part data, the reproduction unit 106 may reproduce the first and second part data at a volume balance between the first and second part data that is specified by the user. For example, the user may specify the volume balance by operating a volume fader for accepting volume settings. In this case, the reproduction unit 106 may reproduce the first and second part data at the volume balance specified by the user. This allows the reproduction unit 106 to reproduce the first and second part data at the volume balance desired by the user.

[0074] Furthermore, when reproducing the first and second part data, the reproduction unit 106 may reproduce the first and second part data with a volume balance determined by a predetermined rule base. For example, the reproduction unit 106 may automatically control the volume balance depending on the type of instrument played by the first part data and the type of instrument played by the second part data. Furthermore, the reproduction unit 106 may automatically control the volume balance depending on the type of instrument played by the first and second parts and the type of instrument played by the user. Furthermore, the reproduction unit 106 may automatically control the volume balance depending on, for example, the user's performance level or user preferences. The predetermined rule may be set by the program creator or by the user. This allows the reproduction unit 106 to output the sounds of each part with an automatically adjusted volume balance in accordance with the performance.

[0075] The playback unit 106 may adjust panning (left / right positioning) instead of or in addition to the volume balance. For example, the playback unit 106 may play back each part data at a left / right positioning specified by the user for each part, or at a left / right positioning determined automatically.

[0076] (3) In the above embodiment, the first part data corresponding to the vocal part and the chord information corresponding to the song data 12b are generated from the song data 12b before the performance data is acquired based on the user's performance operation on the performance device 20. However, the first part data may be extracted and separated from the song data 12b after the user's performance position on the performance device 20 is identified. In this case, the first part data is extracted and separated from the song data 12b after the identified performance position.

[0077] (4) In the above embodiment, the first part data is provided to the speaker 17 so as to be reproduced in accordance with the playing position. Performance data based on the user's performance operation on the performance device 20 may be provided to the speaker 17 along with the first part data. In this case, a sound based on the first part data and a sound based on the performance data may be output from the speaker 17. Alternatively, the first part data may be provided to the performance device 20, the performance data and the first part data may be provided to the speaker 27, and the speaker 27 may output a sound based on the performance data and a sound based on the first part data. In this manner, the playback unit 106 may play both the first part data generated by the generation unit 101 and the performance data acquired by the performance data acquisition unit 103 from the same terminal or the same speaker. As a result of such processing, the performance assistance function 100 can, for example, simultaneously play the user's performance sound and the playback sound of other parts that are synchronized with the user's performance.

[0078] (5) In the above embodiment, the first part data and chord information are generated from the song data 12b stored in the storage unit 12. In the above embodiment, the song data 12b is sound waveform data in WAV format or the like. However, the song data 12b corresponding to a specific song may be generated (obtained) from sound generation control data that specifies sound generation timing, such as MIDI format data or musical score data.

[0079] 8 is a block diagram showing an example of the configuration of a performance assistance function 100A of this modified example. The configuration of the performance assistance function 100A is substantially the same as the configuration of the performance assistance function 100 described with reference to FIG. 4, except that it includes a data conversion unit 108. The data conversion unit 108 generates song data 12b, which is sound waveform data, from sound generation control data that specifies the sound generation timing, such as MIDI format data or musical score data. The song data 12b is generated by converting the sound generation control data into sound waveform data. The conversion from sound generation control data to sound waveform data can be achieved using known conversion technology.

[0080] The song data 12b, which is sound waveform data generated from the sound generation control data by the data conversion unit 108, is supplied to the generation unit 101 and chord analysis unit 102, just like the performance assistance function 100. The subsequent processing is the same as the processing by the performance assistance function 100.

[0081] In this way, the performance assistance function 100A acquires the first part data from the song data 12b, which is sound waveform data generated based on the sound generation control data that defines the sound generation timing. Therefore, the performance assistance function 100A can realize a performance that matches the user's performance using the sound generation control data such as MIDI data.

[0082] If the song data 12b is sound generation control data including chord information, the performance assistance function 100A may extract chord information corresponding to each part from the song data 12b. That is, as long as the performance assistance function 100A acquires chord information for the song to be performed, it may generate the chord information from song data, which is sound waveform data, or extract it from song data, which is sound generation control data. In other words, the performance assistance function 100, 100A only needs to have a functional component (e.g., the generating unit 101) that operates as a part acquisition unit that acquires first part data from the song data 12b, but does not necessarily need to have a functional component that generates or extracts first part data. For example, if the song data 12b is sound generation control data, the performance assistance function 100A does not need to have the generating unit 101, but may acquire part data specified by the user from the part data included in the song data 12b, which is sound generation control data, as first part data and play the acquired first part data.

[0083] (6) In the above-described embodiment, the performance data based on the user's performance operation is sound generation control data such as MIDI, but the performance data is not limited to this and may be sound waveform data. In such a case, the performance position identification unit 104 identifies the performance position by analyzing the sound waveform data acquired as the performance data. To give a more specific example, the performance position identification unit 104 may identify the performance position by comparing the song data 12b, which is sound waveform data, with the performance data, which is sound waveform data. Furthermore, the performance position identification unit 104 may identify the performance position by, for example, comparing the first part data, which is sound waveform data generated by the generation unit 101, with the performance data, which is sound waveform data.

[0084] Furthermore, for example, the performance position identification unit 104 may identify the performance position by comparing the song data 12b, which is sound generation control data, with the performance data, which is sound waveform data. For example, the performance position identification unit 104 may identify the performance position by analyzing the performance data to obtain chord information and comparing the obtained chord information with the song data 12b.

[0085] (7) In the above-described embodiment, chord information is provided as a visually recognizable image. A link to an external site where sheet music data corresponding to the song data 12b can be purchased may be displayed within this image. This link may be, for example, the URL of the external site where sheet music data can be purchased. Furthermore, a song corresponding to the song data 12b can be identified using known song search technology. The URL of the external site where sheet music data for the identified song can be purchased may be, for example, the URL of a site that appears at the top of a search results page when searching for sheet music data for the identified song on a search engine. The URL of the external site where sheet music data can be purchased may be displayed, for example, as a link URL N to the external site in the setting area SA of the screens 500 and 500A shown in FIGS. 5 and 6 .

[0086] (8) In the above-described embodiment, the first part data and chord information are generated from the song data 12b after the performance assistance process is started by a user instruction. However, the first part data and chord information may be generated from the song data 12b in advance and stored in the storage unit 12 before the performance assistance process is started. In this case, when the performance assistance process is started by a user instruction, the process of S303 in FIG. 7 begins, and the processes of S301 and S302 are omitted.

[0087] (9) In the above-described embodiment, the performance assistance system 1 includes the terminal 10 and the performance device 20. However, the performance assistance system 1 can also be realized with only one of these devices. For example, the performance assistance function 100 executed by the terminal 10 may be executed by the control unit 21 of the performance device 20. Furthermore, part or all of the performance assistance function 100 may be realized by an external device, such as a server, capable of communicating with the terminal 10. In this case, the performance assistance process may be executed by the external device, and first part data may be provided to the terminal 10 in accordance with the identified performance position. Furthermore, image data for displaying chord information on the display unit 15 as an image that can be visually recognized by the user may be generated by the external device. The generated image data is provided to the terminal 10.

[0088] (10) In the above-described embodiment, the song data 12b is a single piece of sound waveform data including multiple parts. However, the song data 12b may be composed of multiple pieces of sound waveform data corresponding to the multiple parts, including the vocal part. In this case, when the performance assistance process starts, the generation unit 101 in the performance assistance function 100, 100A acquires the sound waveform data corresponding to the vocal part from the song data 12b as the first part data.

[0089] (11) In the above-described embodiment, the image that allows visual recognition of chord information is displayed on the display unit 15 of the terminal 10. However, this image may be supplied to an external display device that is not included in the configuration of the performance assistance system 1 and displayed on the display unit of that display device.

[0090] (12) In the above-described embodiment, the user's performance position is identified using the trained model 12c. However, the method of identifying the performance position is not limited to using the trained model 12c. It is also possible to identify the user's performance position using existing performance analysis technology. For example, the performance assistance function 100 calculates feature quantities at predetermined time intervals from the song data 12b, which is sound waveform data. The performance assistance function 100 may then calculate feature quantities of the performance data, which is sound waveform data, and compare the feature quantities calculated from the song data 12b with the feature quantities calculated from the performance data to identify the performance position. Alternatively, the performance assistance function 100 may generate sound generation control data from the song data 12b, which is sound waveform data, using a known conversion technology, and compare the sound generation control data with the performance data, which is the performance position, to identify the performance position. Alternatively, the performance assistance function 100 may generate sound generation control data from the performance data, which is sound waveform data, using a known conversion technology, and compare the sound generation control data with the song data 12b, which is the performance data, to identify the performance position.

[0091] The performance assistance function 100 described above uses a trained model 12c obtained by machine learning the correlation between performance data and the chord progression and performance position of a predetermined piece of music. This trained model 12c can be realized by training various known models so that, when performance data, which is sound waveform data or sound generation control data, and chord information analyzed from the song data 12b are input, the trained model 12c outputs information indicating the performance position being played in the performance data. However, the embodiment is not limited to this.

[0092] For example, the performance assistance function 100 may use a trained model 12c that has been trained to output information indicating a performance position from song data 12b, which is sound waveform data, and performance data, which is sound generation control data. More specifically, the performance assistance function 100 may use a trained model 12c that has been trained to output information indicating a performance position being played in the performance data when it receives input of each feature amount or the sound waveform data generated at predetermined time intervals from sound waveform data of a certain song, and the performance data, which is sound generation control data.

[0093] Furthermore, for example, the performance assistance function 100 may use the trained model 12c trained to output information indicating a performance position from the song data 12b, which is sound waveform data, and the performance data, which is sound waveform data. More specifically, the performance assistance function 100 may use the trained model 12c trained to output information indicating a performance position being played in the performance data when it receives feature amounts or the sound waveform data generated at predetermined time intervals from the sound waveform data of a certain song, and feature amounts or performance data generated from the performance data, which is sound waveform data.

[0094] Furthermore, for example, the performance assistance function 100 may use a trained model 12c that has been trained to output information indicating a performance position from the song data 12b, which is sound generation control data, and the performance data, which is sound waveform data. More specifically, the performance assistance function 100 may use a trained model 12c that has been trained to output information indicating a performance position being played in the performance data when the performance assistance function 100 receives input of the sound generation control data of a certain song and feature quantities or performance data generated from the performance data, which is sound waveform data.

[0095] (13) In the above-described embodiment, the performance position identification unit 104 identifies the user's performance position in the song data 12b based on the correspondence between the chord information acquired from the chord analysis unit 102 and the performance data acquired from the performance data acquisition unit 103. However, the embodiment is not limited to this. For example, if chord information for a song has already been registered, the performance assistance function 100 may acquire the registered chord information from the song data 12b instead of the chord information generated by the chord analysis unit 102. Alternatively, the performance assistance function 100 may acquire chord information prepared in advance by a user or the like. To give a more specific example, the performance assistance function 100 may receive chord information input from a user or the like and acquire the received chord information, or may acquire chord information distributed via the Internet or the like. In such a case, the performance assistance function 100 does not need to include the chord analysis unit 102. In this way, the performance assistance function 100 may acquire chord information corresponding to the song data 12b by any means.

[0096] (14) In the above-described embodiment, the generation unit 101 generates first part data corresponding to a first part from song data 12b including multiple parts, and the playback unit 106 plays back the first part data. Here, the generation unit 101 may generate part data for each of the multiple parts included in the song data 12b. In such a case, the playback unit 106 may play back, as the first part data, part data corresponding to a part specified by the user. Alternatively, the generation unit 101 may generate only the first part data corresponding to the part specified by the user from the song data 12b.

[0097] When receiving a specification of a part to be played from the user, the performance assistance function 100 may receive the specification in any manner. For example, the performance assistance function 100 analyzes the song data 12b, which is sound waveform data, to generate first part data, which is sound waveform data, for each predetermined performance means (e.g., musical instrument, vocals, etc.). Alternatively, the performance assistance function 100 analyzes the song data 12b, which is sound waveform data, to generate first part data, which is sound waveform data, for each performance means (e.g., musical instrument, vocals, etc.), and estimates the performance means being played in the generated first part data. The performance assistance function 100 then provides the user with content indicating the predetermined performance means or the estimated performance means, and receives from the user a specification of the performance means to be played. As a result, the performance assistance function 100 can reproduce the first part data corresponding to the specified performance means in sync with the user's performance.

[0098] The performance assistance function 100 may generate part data for each performance means, such as a vocal part including vocals, a drum part including drum sounds, a bass part including bass sounds, and a synth part including synthesizer sounds. For example, by tracking only the vocal part, the performance assistance function 100 can give the user a sense of unity as if the song is being sung in sync with the user's performance, allowing the user to experience a minimal ensemble with the vocal part. The performance assistance function 100 may also acquire first part data corresponding to a drum part including drums from the song data 12b and play the acquired first part data in accordance with the specified performance position. In this way, by tracking only the drum part, the performance assistance function 100 can allow the user to play while feeling the beat (sense of tempo). The performance assistance function 100 can also track all parts other than the part played by the user, allowing the user to feel the unity of a band performance.

[0099] The performance assistance function 100 may also generate part data for each tone color, in addition to each performance means. The performance assistance function 100 may generate part data for only a single performance means or a single tone color, or may generate part data for multiple performance means or multiple tone colors. In this way, the performance assistance function 100 may generate part data at any granularity and play it back in accordance with the user's performance.

[0100] The part designated by the user is not limited to the above-described modification and may be any part. For example, the user may designate a part based on the level of his or her performance skill. In this case, the generation unit 101 generates part data corresponding to the part designated based on the user's performance skill. The playback unit 106 may then play the part data in accordance with the identified performance position. As a result of this processing, the performance assistance function 100 can synchronize (follow) the playback of the part desired by the user to the user's performance, for example, based on the user's performance skill level and the composition of the music being played. To give a more specific example, the performance assistance function 100 may play a drum part while the user is still inexperienced in playing, allowing the user to acquire a sense of tempo. Once the user becomes more accustomed to playing, the performance assistance function 100 may play only the vocal part, allowing the user to experience a minimal ensemble with the vocal part. The performance assistance function 100 may determine the user's performance skill and automatically generate first part data corresponding to a part based on the determined level of performance skill, or may suggest the first part data to the user as a part to follow.

[0101] The user may also specify a part according to the composition of the musical piece the user wishes to play. In this case, the generating unit 101 generates part data corresponding to the part specified according to the composition of the musical piece the user wishes to play. The playing unit 106 may then play the part data in accordance with the identified performance position. The performance assistance function 100 may determine the composition of the musical piece the user wishes to play, and automatically generate first part data corresponding to a part according to the determined composition of the musical piece, or may suggest the first part data to the user as a part to be followed.

[0102] (15) Of the processes described in the above embodiments as being performed manually, all or part of the processes can also be performed automatically using known methods. Furthermore, the information, including the process procedures, specific names, various data, and parameters shown in the above text and drawings, can be changed as desired unless otherwise specified. For example, the various information shown in each figure is not limited to the illustrated information. Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown in the drawings. All or part of the devices can be functionally or physically distributed and integrated in any unit depending on various loads or usage conditions. Furthermore, the above-described embodiments and variations can be combined as appropriate to the extent that the processing content is not inconsistent.

[0103] Although several embodiments of the present application have been described in detail above with reference to the drawings, these are merely examples, and the present invention can be implemented in other forms, including those described in the Disclosure of the Invention section, with various modifications and improvements based on the knowledge of those skilled in the art. Furthermore, the performance assistance function 100 described above can be flexibly configured, for example, by calling an external platform, etc., via an API (Application Programming Interface) or network computing, depending on the function. Furthermore, the term "unit" in the claims and specification can be read as "means" or "circuit," etc. For example, a "generation unit" can be read as a "generation means" or a "generation circuit."

[0104] 1: Performance assistance system, 10: terminal, 11: control unit, 12: memory unit, 12a: program, 12b: song data, 12c: trained model, 13: communication unit, 14: operation unit, 15: display unit, 16: interface, 17: speaker, 18: bus, 20: performance device, 21: control unit, 22: memory unit, 22a: program, 23: sound source unit, 24: performance operator, 25: sensor, 26: interface, 27: speaker, 28: bus, 100, 100A: performance assistance function, 101: generation unit, 102: chord analysis unit, 103: performance data acquisition unit, 104: performance position identification unit, 105: provision unit, 106: playback unit, 107: judgment unit, 108: data conversion unit, 500, 500A: screen

Claims

1. A performance assistance method implemented by a computer, comprising: a performance data acquisition step of acquiring performance data of a performance by a user; a part acquisition step of acquiring first part data corresponding to a predetermined first part from song data including multiple parts; an identification step of identifying a performance position of the performance by the user based on the correspondence between the performance data and the song data; and a playback step of playing back the first part data in accordance with the identified performance position.

2. The musical performance assistance method according to claim 1, wherein the part acquisition step acquires the first part data corresponding to the first part different from the performance part played by the user.

3. The performance assistance method according to claim 1, wherein the part acquisition step further acquires second part data corresponding to a second part different from the first part from the song data, and the reproduction step reproduces the first part data and the second part data in accordance with the identified performance position.

4. The musical performance assistance method according to claim 3, wherein the reproducing step reproduces the first part data and the second part data with a volume balance designated by the user.

5. The musical performance assistance method according to claim 3, wherein the reproducing step reproduces the first part data and the second part data with a volume balance determined by a predetermined rule.

6. The performance assistance method according to claim 1, wherein the part acquisition step acquires part data corresponding to one or more parts selected by the user from the song data as the first part data, and the reproduction step reproduces the first part data corresponding to the part selected by the user in accordance with the identified performance position.

7. The performance assistance method according to claim 1, further comprising: a chord acquisition step of acquiring chord information corresponding to the song data; wherein the identification step identifies a performance position of the performance by the user based on the correspondence between the chord information and the performance data.

8. The performance assistance method according to claim 1, further comprising: a chord acquisition step of acquiring chord information corresponding to said song data; and a provision step of providing said chord information in a visually recognizable manner.

9. A performance assistance method according to claim 8, wherein the chord acquisition step generates information including a chord string in which information indicating chords is arranged in chronological order as the chord information, and the provision step provides a chord in the chord string that corresponds to the performance position in a visually recognizable manner.

10. The performance assistance method according to claim 9, wherein the providing step provides the next chord to the chord corresponding to the performance position in a visually recognizable manner.

11. The performance assistance method according to claim 1, wherein the identifying step identifies the playing position of the performance by the user using a trained model that has learned the correlation between performance data and song data.

12. A performance assistance method according to claim 11, wherein the identifying step identifies the playing position of the performance by the user using, as the trained model, a trained model that has learned the correlation between performance data of a predetermined song and the chord progression in the song data and the playing position in the performance data.

13. The performance assistance method according to claim 1, wherein the part acquisition step acquires the first part data from the song data, which is sound waveform data generated based on sound generation control data that defines sound generation timing.

14. The performance assistance method according to claim 1, wherein the part acquisition step acquires the first part data from the song data, which is sound waveform data.

15. The performance assistance method according to claim 1, wherein the reproduction step reproduces the first part data and the performance data based on the performance from the same speaker.

16. The musical performance assistance method according to claim 1, wherein the part acquisition step acquires the first part data corresponding to a vocal part including vocals.

17. The performance assistance method according to claim 1, wherein the part acquisition step acquires the first part data corresponding to a drum part including a drum.

18. A performance assistance system comprising: a performance data acquisition unit that acquires performance data of a performance by a user; a part acquisition unit that acquires first part data corresponding to a first part different from the performance part played by the user from song data including multiple parts; an identification unit that identifies a performance position of the performance by the user based on the correspondence between the performance data and the song data; and a playback unit that plays back the first part data in accordance with the identified performance position.

19. A program for causing a computer to execute the performance assistance method described in any one of claims 1 to 17.

Images