Reproduction control method and reproduction control system
The method and system allow for precise control of sound intervals in music reproduction by continuing the playback of a first sound until its end and starting the second sound upon user instruction, addressing the challenge of synchronizing with performer intentions.
Patent Information
- Application Number
- JP2020173419
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-07-31
- Filing Date
- 2020-10-14
- Publication Date
- 2024-11-06
- Estimated Expiration
- 2040-10-14
AI Technical Summary
Existing technologies struggle to appropriately control the interval between two notes in music reproduction due to variations in performers' intentions or preferences, particularly during pauses or fermatas, making it difficult to synchronize the performance with the reproduction accurately.
A method and system that uses sound data to control the reproduction of music by continuing the playback of a first sound until its end point and then starting the playback of a second sound immediately after a user-generated instruction, allowing for variable intervals based on user input.
This approach enables precise control over the interval between sounds, aligning the reproduction with the performer's intentions and preferences, ensuring synchronized and expressive musical performance.
Smart Images

Figure 0007577964000001 
Figure 0007577964000002 
Figure 0007577964000003
Abstract
Description
[Technical field]
[0001] The present disclosure relates to techniques for controlling sound reproduction. [Background technology]
[0002] There have been proposed techniques for making the playback of a piece of music follow the performance of a performer. For example, Patent Document 1 discloses a technique for estimating the performance position in a piece of music by analyzing musical tones generated by the performance of the piece of music, and controlling the automatic performance of the piece of music according to the estimation result. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2017-207615 A Summary of the Invention [Problem to be solved by the invention]
[0004] A performer may pause during the performance of a piece of music to reflect the performer's intention (e.g., musical expression) or a performance mark such as a fermata. The length of the pause period may vary from performer to performer or from performance to performance by a performer, depending on the performer's intention or preference. Therefore, in playing a piece of music, it is difficult to appropriately control the interval between two sounds located before and after the pause period, depending on the performer's intention or preference. In consideration of the above circumstances, one of the objects of one embodiment of the present invention is to appropriately control the interval between two adjacent sounds in playing a plurality of sounds. [Means for solving the problem]
[0005] In order to solve the above problems, a playback control method according to one embodiment of the present disclosure is a method of playing a plurality of sounds using sound data representing a time series of the plurality of sounds, in which a first sound among the plurality of sounds that is being played at the time a first instruction is issued is continued until an end point of the first sound represented by the sound data, and after playback of the first sound is stopped, playback of a second sound among the plurality of sounds that is immediately after the first sound is started in response to a second instruction from a user.
[0006] A playback control method according to another aspect of the present disclosure is a method of playing a plurality of sounds using sound data representing a time series of the plurality of sounds that constitute a piece of music, the method estimating the point in the piece of music at which a user is playing the plurality of sounds in parallel with the playback of the plurality of sounds, and causing the playback of the plurality of sounds to follow the performance of the piece of music depending on the result of the estimation, and in playing the plurality of sounds, playing a first sound among the plurality of sounds that corresponds to a first instruction, and after playback of the first sound has stopped, starting playback of a second sound among the plurality of sounds that is immediately after the first sound in response to a second instruction from the user.
[0007] A playback control method according to another aspect of the present disclosure is a method of playing back a plurality of sounds using sound data representing a time series of the sounds, wherein a first sound among the plurality of sounds that is being played at the time a first instruction is issued is continued until an end point of the first sound represented by the sound data. [Brief description of the drawings]
[0008] [Figure 1] 1 is a block diagram illustrating the configuration of a playback system according to a first embodiment. [Diagram 2] FIG. 2 is a schematic diagram of music data. [Diagram 3] 2 is an explanatory diagram of the configuration and state of an operating device. FIG. [Figure 4] FIG. 2 is a block diagram illustrating a functional configuration of the playback control system. [Diagram 5] FIG. 2 is an explanatory diagram of the relationship between the playback of a playback part by a playback device and a first instruction and a second instruction. [Figure 6]10 is a flowchart illustrating a specific procedure of a playback control process. [Figure 7] 13 is an explanatory diagram relating to a state of an operating device in a second embodiment. FIG. [Figure 8] 13 is an explanatory diagram relating to a state of an operating device in a third embodiment. FIG. [Figure 9] FIG. 13 is a block diagram illustrating a functional configuration of a playback control system according to a fourth embodiment. [Figure 10] FIG. 4 is an explanatory diagram of the operation of the edit processing unit. [Figure 11] 11 is a flowchart illustrating a specific procedure of an editing process. [Figure 12] FIG. 13 is a block diagram illustrating the configuration of a playback system in a fifth embodiment. [Figure 13] FIG. 13 is a block diagram illustrating a functional configuration of a playback control system in a fifth embodiment. [Figure 14] 11 is a flowchart illustrating a specific procedure of a reference data generation process. [Figure 15] FIG. 13 is a schematic diagram of a preparation screen displayed in parallel with the reference data generation process. [Figure 16] FIG. 13 is a schematic diagram of a playback screen displayed in parallel with the playback control process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] A: First embodiment FIG. 1 is a block diagram illustrating the configuration of a playback system 100 according to the first embodiment. The playback system 100 is installed in an acoustic space where a user U is present. The user U is a performer who plays a specific part of a piece of music (hereinafter referred to as a "performance part") using a musical instrument 200 such as a string instrument. The playback system 100 is a computer system that plays the piece of music in parallel with the performance of the performance part by the user U. Specifically, the playback system 100 plays a part other than the performance part (hereinafter referred to as a "playback part") among a plurality of parts that make up a piece of music. The performance part is, for example, a part that makes up the main melody of the piece of music. The playback part is, for example, one or more parts that make up the accompaniment of the piece of music. As can be understood from the above explanation, the performance of the performance part by the user U and the playback of the playback part by the playback system 100 are executed in parallel to realize the performance of the piece of music. The performance part and the playback part may be a common part of the piece of music.
[0010] The playback system 100 includes a playback control system 10 and a playback device 20. The playback control system 10 and the playback device 20 are configured separately and communicate with each other by wire or wirelessly. Note that the playback control system 10 and the playback device 20 may be configured as an integrated unit.
[0011] The playback device 20 plays the playback part of the music piece under the control of the playback control system 10. Specifically, the playback device 20 is an automatic performance instrument that executes automatic performance of the playback part. For example, an automatic performance instrument (e.g., an automatic performance piano) different from the instrument 200 played by the user U is used as the playback device 20. As can be understood from the above explanation, automatic performance is one aspect of "playback".
[0012] The playback device 20 of the first embodiment includes a drive mechanism 21 and a sound generation mechanism 22. The sound generation mechanism 22 is a mechanism that generates musical tones. Specifically, the sound generation mechanism 22 includes a string-striking mechanism for each key that generates sound from a string (sound source) in response to the displacement of each key on the keyboard, similar to a natural musical instrument keyboard. The drive mechanism 21 drives the sound generation mechanism 22 to perform automatic performance of a musical piece. The drive mechanism 21 drives the sound generation mechanism 22 in response to an instruction from the playback control system 10, thereby realizing automatic performance of the playback part.
[0013] The playback control system 10 is a computer system that controls the playback of the playback part by the playback device 20, and includes a control device 11, a storage device 12, a sound collection device 13, and an operation device 14. The playback control system 10 is realized by, for example, a portable terminal device such as a smartphone or a tablet terminal, or a portable or stationary terminal device such as a personal computer. The playback control system 10 may be realized by a single device, or may be realized by multiple devices configured separately from each other.
[0014] The control device 11 is a single or multiple processors that control each element of the playback control system 10. Specifically, the control device 11 is configured by one or more types of processors, such as a CPU (Central Processing Unit), an SPU (Sound Processing Unit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or an ASIC (Application Specific Integrated Circuit).
[0015] The storage device 12 is a single or multiple memories that store the programs executed by the control device 11 and various data used by the control device 11. The storage device 12 is configured with a known recording medium such as a magnetic recording medium or a semiconductor recording medium, or a combination of multiple types of recording media. In addition, a portable recording medium that is detachable from the playback control system 10, or a recording medium that can be written to or read from via a communication network (e.g., cloud storage) may be used as the storage device 12.
[0016] The storage device 12 stores, for each piece of music, music data M that specifies the time series of multiple notes that make up the piece of music. FIG. 2 is a schematic diagram of the music data M. The music data M includes reference data R and performance data D. The reference data R specifies the time series of notes in the performance part that the user U plays. Specifically, the reference data R specifies the pitch and sound duration for each of the multiple notes in the performance part. On the other hand, the performance data D specifies the time series of notes in the playback part that the playback device 20 plays. Specifically, the performance data D specifies the pitch and sound duration for each of the multiple notes in the playback part. Each of the reference data R and the performance data D is time-series data in a Musical Instrument Digital Interface (MIDI) format in which, for example, instruction data that instructs the sounding or muting of a musical tone and time data that specifies the time point of the action instructed by the instruction data are arranged in time series. The instruction data specifies, for example, the pitch and intensity to instruct an action such as sounding or muting. The time data specifies, for example, the interval between adjacent instruction data. The period from when instruction data instructs the sounding of a particular pitch to when subsequent instruction data instructs the muting of that pitch is the sounding period for the note of that pitch. The performance data D is an example of "sound data" that represents a time series of multiple sounds. The multiple notes of the playback part represented by the performance data D are an example of "multiple sounds" represented by "sound data".
[0017] The sound collection device 13 in FIG. 1 collects musical sounds produced by the musical instrument 200 when played by the user U, and generates an audio signal Z representing the waveform of the musical sound. For example, a microphone is used as the sound collection device 13. For convenience, an A / D converter that converts the audio signal Z generated by the sound collection device 13 from analog to digital is omitted from the illustration. In the first embodiment, a configuration in which the sound collection device 13 is mounted on the playback control system 10 is illustrated, but the sound collection device 13 that is separate from the playback control system 10 may be connected to the playback control system 10 by wire or wirelessly. In addition, the playback control system 10 may receive an output signal from an electric musical instrument such as an electric string instrument as the audio signal Z. As can be understood from the above explanation, the sound collection device 13 may be omitted from the playback control system 10.
[0018] The operation device 14 is an input device that accepts instructions from the user U. As illustrated in FIG. 3, the operation device 14 of the first embodiment includes a movable part 141 that moves when operated by the user U. The movable part 141 is an operation pedal that the user U can operate with his / her foot. For example, a pedal-type MIDI controller is used as the operation device 14. The user U can operate the operation device 14 at a desired time in parallel with playing the musical instrument 200 while playing the musical instrument 200 with both hands. Note that a touch panel that detects contact by the user U may be used as the operation device 14.
[0019] The operating device 14 transitions from one of the released state and the operating state to the other in response to an operation by the user U. The released state is a state in which the operating device 14 is not operated by the user U. Specifically, the released state is a state in which the user U does not step on the movable part 141. The released state is also expressed as a state in which the movable part 141 is at position H1. On the other hand, the operating state is a state in which the operating device 14 is operated by the user U. Specifically, the operating state is a state in which the user U steps on the movable part 141. The operating state is also expressed as a state in which the movable part 141 is at position H2 different from position H1. The released state is an example of a "first state", and the operating state is an example of a "second state".
[0020] 4 is a block diagram illustrating an example of the functional configuration of the playback control system 10. The control device 11 executes a program stored in the storage device 12 to realize a plurality of functions (a performance analysis unit 31, a playback control unit 32, and an instruction receiving unit 33) for controlling the playback of the playback part by the playback device 20.
[0021] The performance analysis unit 31 estimates a performance position X in a piece of music by analyzing an audio signal Z supplied from the sound collection device 13. The performance position X is the point in time in the piece of music where the user U is currently performing. The estimation of the performance position X is repeatedly performed in parallel with the performance of the performance part by the user U and the reproduction of the reproduction part by the reproduction device 20. That is, the performance position X is estimated at each of a plurality of points on the time axis. The performance position X moves backward in the piece of music as time passes.
[0022] Specifically, the performance analysis unit 31 calculates the playing position X by comparing the reference data R of the music data M with the audio signal Z. Any known analysis technique (score alignment technique) may be adopted for the estimation of the playing position X by the performance analysis unit 31. For example, the analysis technique disclosed in Japanese Patent Application Laid-Open No. 2016-099512 is used for estimating the playing position X. The performance analysis unit 31 may also estimate the playing position X by using a statistical estimation model such as a deep neural network or a hidden Markov model.
[0023] The playback control unit 32 causes the playback device 20 to play back each note specified by the performance data D. That is, the playback control unit 32 causes the playback device 20 to automatically play the playback part. Specifically, the playback control unit 32 moves a position Y in the music piece to be played back (hereinafter referred to as the "playback position") backward over time, and sequentially supplies instruction data in the performance data D that corresponds to the playback position Y to the playback device 20. That is, the playback control unit 32 functions as a sequencer that sequentially supplies each instruction data included in the performance data D to the playback device 20. The process by which the playback control unit 32 causes the playback device 20 to play back the playback part is executed in parallel with the performance of the performance part by the user U.
[0024] The playback control unit 32 makes the playback of the playback part by the playback device 20 follow the performance of the music piece by the user U according to the result of the estimation of the performance position X by the performance analysis unit 31. That is, the automatic performance of the playback part by the playback device 20 proceeds at the same tempo as the performance of the performance part by the user U. For example, when the progress of the performance position X (i.e., the performance speed by the user U) is fast, the playback control unit 32 increases the speed of progress of the playback position Y (the playback speed by the playback device 20), and when the progress of the performance position X is slow, the playback control unit 32 decreases the speed of progress of the playback position Y. That is, the automatic performance of the playback part is executed at the same performance speed as the performance by the user U so as to be synchronized with the progress of the performance position X. Therefore, the user U can play the performance part with the feeling that the playback device 20 is playing the playback part in accordance with his / her performance.
[0025] As described above, in the first embodiment, the playback of multiple notes in the playback part follows the performance of the instrument 200 by the user U, so that the intentions (e.g., performance expression) or preferences of the user U can be appropriately reflected in the playback of the playback part.
[0026] The instruction receiving unit 33 receives a first instruction Q1 and a second instruction Q2 from the user U. The first instruction Q1 and the second instruction Q2 are generated by the user U's operation on the operation device 14. The first instruction Q1 is an instruction to temporarily stop the playback of the playback part by the playback device 20. The second instruction Q2 is an instruction to resume the playback of the playback part stopped by the first instruction Q1.
[0027] Specifically, the instruction receiving unit 33 receives an operation by the user U to transition the operation device 14 from a released state to an operating state as the first instruction Q1. That is, the user U gives the first instruction Q1 to the playback control system 10 by stepping on the movable part 141 of the operation device 14. For example, the instruction receiving unit 33 specifies the time when the movable part 141 starts moving from position H1 (released state) to position H2 (operated state) as the time of the first instruction Q1. Note that a configuration in which the instruction receiving unit 33 specifies the time when the movable part 141 reaches a position midway between position H1 and position H2 as the time of the first instruction Q1, or a configuration in which the instruction receiving unit 33 specifies the time when the movable part 141 reaches position H2 as the time of the first instruction Q1 is also assumed.
[0028] The instruction receiving unit 33 also receives an operation by the user U to transition the operating device 14 from an operating state to a released state as the second instruction Q2. That is, the user U gives the second instruction Q2 to the playback control system 10 by releasing the movable part 141 of the operating device 14 from a state where the movable part 141 is pressed. For example, the instruction receiving unit 33 specifies the time when the movable part 141 starts moving from position H2 (operated state) to position H1 (released state) as the time of the second instruction Q2. Note that a configuration in which the instruction receiving unit 33 specifies the time when the movable part 141 reaches a position midway between position H2 and position H1 as the time of the second instruction Q2, or a configuration in which the instruction receiving unit 33 specifies the time when the movable part 141 reaches position H1 as the time of the second instruction Q2 is also assumed.
[0029] The user U can give the first instruction Q1 and the second instruction Q2 at any time during the performance of the performance part. Therefore, the interval between the first instruction Q1 and the second instruction Q2 is variable in length according to the intention of the user U. For example, the user U gives the first instruction Q1 before the start of a rest period in the music piece, and gives the second instruction Q2 when the rest period of the user U's desired length has elapsed.
[0030] 5 is an explanatory diagram of the relationship between the reproduction of the reproduction part by the reproduction device 20 and the first instruction Q1 and the second instruction Q2. The sounding period of each note specified by the performance data D and the sounding period of each note actually reproduced by the reproduction device 20 are shown together in FIG.
[0031] The note N1 in FIG. 5 is a note corresponding to the first instruction Q1 among the multiple notes specified by the performance data D. Specifically, the note N1 is a note being reproduced by the reproduction device 20 at the time of the first instruction Q1 among the multiple notes in the reproduction part. After the first instruction Q1 is generated, the reproduction control unit 32 causes the reproduction device 20 to continue reproduction of the note N1 until the end point of the sounding period specified for the note N1 by the performance data D. For example, the reproduction control unit 32 supplies the reproduction device 20 with instruction data for instructing the mute of the note N1 at the end point of the sounding period of the note N1. As can be understood from the above description, the reproduction of the note N1 is not immediately stopped at the time of the first instruction Q1, but is continued until the end point specified by the performance data D even after the generation of the first instruction Q1. Note that the note N1 is an example of a "first note."
[0032] The note N2 in FIG. 5 is a note immediately after the note N1 among the multiple notes specified by the performance data D. After the playback of the note N1 is stopped, the playback control unit 32 causes the playback device 20 to start playing the note N2 in response to the second instruction Q2 from the user U. That is, regardless of the position of the start point of the sounding period specified for the note N2 by the performance data D and the time length of the interval between the note N1 and the note N2 specified by the performance data D, the playback of the note N2 is started on the condition that the second instruction Q2 is generated. Specifically, when the instruction receiving unit 33 receives the second instruction Q2, the playback control unit 32 supplies the instruction data of the note N2 in the performance data D to the playback device 20. Therefore, the playback of the note N2 is started immediately after the second instruction Q2. The note N2 is an example of the "second note."
[0033] 6 is a flowchart illustrating a specific procedure of an operation (hereinafter referred to as a "playback control process") Sa in which the control device 11 controls the playback device 20. In response to an instruction from the user U, the playback control process Sa is started.
[0034] When the playback control process Sa is started, the control device 11 judges whether the standby data W is in a valid state (Sa1). The standby data W is data (e.g., a flag) that indicates that the playback of the playback part is temporarily stopped by the first instruction Q1, and is stored in the storage device 12. Specifically, the standby data W is set to a valid state (e.g., W=1) when the first instruction Q1 is generated, and is set to an invalid state (e.g., W=0) when the second instruction Q2 is generated. The standby data W can also be said as data that indicates a state of waiting for the playback of the playback part to be resumed.
[0035] If the standby data W is not valid (Sa1: NO), the control device 11 (performance analysis unit 31) estimates the performance position X by analyzing the audio signal Z supplied from the sound collection device 13 (Sa2). The control device 11 (playback control unit 32) causes the playback device 20 to proceed with the playback of the playback part according to the result of the estimation of the performance position X (Sa3). That is, the control device 11 controls the playback of the playback part by the playback device 20 so as to follow the performance of the performance part by the user U.
[0036] The control device 11 (instruction receiving unit 33) judges whether or not the first instruction Q1 has been received from the user U (Sa4). When the first instruction Q1 has been received (Sa4: YES), the control device 11 (playback control unit 32) causes the playback device 20 to continue playing the note N1 that is being played at the time of receiving the first instruction Q1 until the end point of the sounding period specified by the performance data D (Sa5). Specifically, the control device 11 advances the playback position Y at a speed (tempo) equal to that at the time when the first instruction Q1 was generated, and when the playback position Y reaches the end point of the sounding period of the note N1, supplies the playback device 20 with instruction data representing the muting of the note N1. After executing the above process, the control device 11 changes the standby data W from an invalid state to a valid state (W=1) (Sa6). Note that the standby data W may be updated (Sa6) before executing step Sa5.
[0037] When the standby data W is set to a valid state, the result of the judgment in step Sa1 becomes positive. When the standby data W is in a valid state (Sa1: YES), the estimation of the playing position X (Sa2), the playback control of the playing part (Sa3), and the processing related to the note N1 (Sa4-Sa6) are not executed. That is, the playback control of the playing part linked to the playing position X is stopped in response to the first instruction Q1 from the user U. Also, when the first instruction Q1 is not accepted (Sa4: NO), the processing related to the note N1 (Sa5, Sa6) is not executed.
[0038] The control device 11 (instruction receiving unit 33) judges whether or not the second instruction Q2 has been received from the user U (Sa7). When the second instruction Q2 has been received (Sa7: YES), the control device 11 (playback control unit 32) causes the playback device 20 to play the note N2 immediately following the note N1 (Sa8). Specifically, the control device 11 updates the playback position Y to the start point of the note N2. That is, the playback of the playback part stopped by the first instruction Q1 is resumed by the second instruction Q2. The control device 11 changes the standby data W from a valid state to an invalid state (W=0) (Sa9). As described above, when the standby data W is set to an invalid state, the result of the judgment in step Sa1 becomes negative. Therefore, the second instruction Q2 is used as a trigger to resume the estimation of the performance position X (Sa2) and the playback control of the playback part (Sa3). Note that the update of the standby data W (Sa8) may be executed before the execution of step Sa8.
[0039] The control device 11 judges whether or not to end the playback of the playback part by the playback device 20 (Sa10). For example, when playback of the playback part is completed to the end point, or when an end command is received from the user U, the control device 11 judges to end the playback of the playback part. If playback of the playback part is not to be ended (Sa10: NO), the control device 11 shifts the process to step Sa1 and repeats the above-exemplified processes (Sa1-Sa9). On the other hand, when the control device 11 judges that playback of the playback part is to be ended (Sa10: YES), the playback control process Sa is ended.
[0040] As described above, in the first embodiment, the note N1 corresponding to the first instruction Q1 is played, and after the playback of the note N1 is stopped, the playback of the note N2 immediately following the note N1 is started in response to the second instruction Q2 by the user U. Therefore, the interval between the playback of the note N1 and the playback of the note N2 (for example, the duration of the rest period in the music piece) can be changed according to each point in time of the first instruction Q1 and the second instruction Q2.
[0041] Furthermore, in the first embodiment, the playback of the note N1 being played at the time of the issuance of the first instruction Q1 continues even after the first instruction Q1 is generated until the end point of the note N1 specified by the performance data D. Therefore, compared to a configuration in which the playback of the note N1 is stopped at the time of the issuance of the first instruction Q1, it is possible to appropriately continue the playback of the note N1 according to the content of the performance data D.
[0042] In the first embodiment, the user U can change the interval between the notes N1 and N2 to an appropriate time length according to the user U's intention or preference by operating the operation device 14. In particular, in the first embodiment, the first instruction Q1 is generated by transitioning the operation device 14 from a released state to an operating state, and after the operating state is maintained, the second instruction Q2 is generated by transitioning the operation device 14 from the operating state to the released state at a desired time point after the generation of the first instruction Q1. That is, the first instruction Q1 and the second instruction Q2 are generated by a series of operations that transition the operation state from the released state to the operating state and then transition it back to the released state. Therefore, the operation of the user U on the operation device 14 is simplified compared to a configuration in which an operation of transitioning the operation device 14 from the released state to the operating state is necessary for each of the first instruction Q1 and the second instruction Q2.
[0043] B: Second embodiment A second embodiment will be described. In each of the following exemplary embodiments, elements having the same functions as those in the first embodiment will be denoted by the same reference numerals as those used in the description of the first embodiment, and detailed descriptions thereof will be omitted as appropriate.
[0044] In the first embodiment, when the first instruction Q1 is generated, the playback position Y is advanced at the same speed as that at the time of the first instruction Q1, and when the playback position Y reaches the end point of the musical note N1, the playback of the musical note N1 is stopped. The playback control unit 32 of the second embodiment variably controls the advancing speed of the playback position Y (i.e., the playback speed of the playback part) according to the operation speed V1 of the movable part 141 after the generation of the first instruction Q1. The operation speed V1 is the speed at which the movable part 141 moves from the position H1 corresponding to the released state toward the position H2 corresponding to the operated state. For example, the average value of a plurality of speeds calculated during the period from the position H1 to the position H2 where the movable part 141 moves is the operation speed V1.
[0045] FIG. 7 is an explanatory diagram regarding the state of the operation device 14 in the second embodiment. As illustrated in FIG. 7, the instruction reception unit 33 receives the first instruction Q1 at the time when the movable part 141 starts to move from the position H1 toward the position H2. The playback control unit 32 controls the advancing speed of the playback position Y according to the operation speed V1 of the movable part 141 after the generation of the first instruction Q1.
[0046] Specifically, the playback control unit 32 increases the advancing speed of the playback position Y as the operation speed V1 is higher. For example, as illustrated in FIG. 7, the advancing speed of the playback position Y when the operation speed V1 is the speed V1_H is higher than the advancing speed of the playback position Y when the operation speed V1 is the speed V1_L (V1_L < V1_H). Therefore, the longer the operation speed V1 is, the shorter the continuation length of the musical note N1 is. For example, the continuation length of the musical note N1 when the operation speed V1 is the speed V1_H is shorter than the continuation length of the musical note N1 when the operation speed V1 is the speed V1_L.
[0047] The second embodiment also achieves the same effect as the first embodiment. In the second embodiment, the duration of the note N1 is controlled according to the operation speed V1, so that the user U can adjust the duration of the note N1. In addition, in the second embodiment, the operation device 14 for giving the first instruction Q1 and the second instruction Q2 is also used for adjusting the duration of the note N1. Therefore, compared with a configuration in which the user U operates separate devices for giving the first instruction Q1 and the second instruction Q2 and for adjusting the duration of the note N1, the operation by the user U is simplified.
[0048] C: Third embodiment In the first embodiment, the playback of the note N2 is started immediately after the second instruction Q2. In the second embodiment, the time from the second instruction Q2 to the start of playback of the note N2 (hereinafter referred to as the "delay time") is variably controlled according to the operation speed V2. The operation speed V2 is the speed at which the movable part 141 moves from the position H2 corresponding to the operated state to the position H1 corresponding to the released state. For example, the average value of multiple velocities calculated within the period until the movable part 141 moves from the position H2 to the position H1 is the operation speed V2.
[0049] Fig. 8 is an explanatory diagram regarding the state of the operation device 14 in the third embodiment. As illustrated in Fig. 8, the instruction receiving unit 33 receives a second instruction Q2 at the time when the movable part 141 starts to move from position H2 to position H1. The playback control unit 32 variably controls the delay time according to the operation speed V2.
[0050] Specifically, the playback control unit 32 shortens the delay time as the operation speed V2 becomes faster. For example, as illustrated in Fig. 8, the delay time when the operation speed V2 is a speed V2_L is longer than the delay time when the operation speed V2 is a speed V2_H (V2_H>V2_L). Therefore, the slower the operation speed V2 is, the later the time point on the time axis at which the playback of the note N2 starts.
[0051] The third embodiment also achieves the same effect as the first embodiment. In the third embodiment, the time when the reproduction of the note N2 starts is controlled according to the operation speed V2, so that the user U can adjust the start point of the first note N2 after the reproduction part is resumed. In addition, in the third embodiment, the operation device 14 for giving the first instruction Q1 and the second instruction Q2 is also used for adjusting the start point of the note N2. Therefore, compared with a configuration in which the user U operates separate devices for giving the first instruction Q1 and the second instruction Q2 and adjusting the start point of the note N2, there is also an advantage that the operation by the user U is simplified. The configuration of the second embodiment may be applied to the third embodiment.
[0052] D: Fourth embodiment FIG. 9 is a block diagram illustrating the functional configuration of the playback control system 10 in the fourth embodiment. The control device 11 in the fourth embodiment functions as an edit processing unit 34 in addition to the same elements as in the first embodiment (performance analysis unit 31, playback control unit 32, and instruction receiving unit 33). The edit processing unit 34 edits the performance data D stored in the storage device 12 in response to an instruction from a user U. The operations of the elements other than the edit processing unit 34 are the same as in the first embodiment. Therefore, the fourth embodiment also achieves the same effects as the first embodiment. The configuration of the second or third embodiment may be applied to the fourth embodiment.
[0053] Fig. 10 is an explanatory diagram of the operation of the edit processing unit 34. Fig. 10 shows notes N1 and N2 specified by the performance data D of the playback part. As in each of the above-mentioned forms, the first instruction Q1 and the second instruction Q2 are generated at any time of the user U. Therefore, a time difference L occurs between the start point of note N2 specified by the performance data D and the time of the second instruction Q2. The edit processing unit 34 edits the performance data D so as to reduce the time difference L.
[0054] 11 is a flow chart illustrating a specific procedure of a process (hereinafter referred to as "edit process") Sb in which the edit processing unit 34 edits the performance data D. For example, the edit process Sb is executed every time the playback of the playback part by the playback device 20 (the above-mentioned playback control process Sa) is repeated a predetermined number of times. Note that the edit process Sb may be started in response to an instruction from the user U.
[0055] When the edit process Sb is started, the edit processing unit 34 calculates the degree of dispersion Δ of the time differences L in a predetermined number of past playback control processes Sa (Sb1). The degree of dispersion Δ is a statistical quantity that represents the degree of dispersion regarding a plurality of time differences L. For example, the variance, standard deviation, or distribution range of the plurality of time differences L is used as the degree of dispersion Δ.
[0056] The edit processing unit 34 judges whether the degree of dispersion Δ exceeds the threshold value Δth (Sb2). If the degree of dispersion Δ exceeds the threshold value Δth, it is presumed that the user U is practicing playing the music while consciously varying the waiting time until the playback of the note N2 is resumed. Therefore, it is not appropriate to edit the performance data D according to the multiple time differences L. On the other hand, if the degree of dispersion Δ is below the threshold value Δth, it is presumed that the multiple time differences L are values that are in line with the intention or preference of the user U (i.e., preferred values unique to the user U).
[0057] Considering the above tendency, when the dispersion degree Δ is below the threshold value Δth (Sb2: NO), the edit processing unit 34 edits (Sb3-Sb4) the performance data D according to the multiple time differences L. On the other hand, when the dispersion degree Δ is above the threshold value Δth (Sb2: YES), the edit processing unit 34 ends the edit process Sb without editing the performance data D (Sb3, Sb4).
[0058] In editing the performance data D, the edit processing unit 34 calculates an average time difference La by averaging a plurality of time differences L (Sb3). Then, the edit processing unit 34 changes the start point of the note N2 specified by the performance data D by the average time difference La (Sb4). For example, when the average time difference La is a negative number, the edit processing unit 34 moves the start point of the note N2 specified by the performance data D forward by the time equivalent to the average time difference La. Also, when the average time difference La is a positive number, the edit processing unit 34 moves the start point of the note N2 specified by the performance data D backward by the time equivalent to the average time difference La. That is, when the user U tends to secure a sufficient waiting time immediately before the note N2, the start point of the note N2 specified by the performance data D is changed backward, and when the waiting time tends to be short, the start point of the note N2 specified by the performance data D is changed forward.
[0059] As can be understood from the above description, in the fourth embodiment, the performance data D is edited according to the time difference L in the performance of the performance parts by the user U. Therefore, it is possible to reflect the tendencies unique to each user U in the performance data D.
[0060] E: Fifth embodiment FIG. 12 is a block diagram illustrating the configuration of a playback system 100 according to the fifth embodiment. The playback system 100 includes a playback control system 10 and a performance device 50. The playback control system 10 of the fifth embodiment includes a display device 15 in addition to the same elements as the playback control system 10 of the first embodiment (a control device 11, a storage device 12, a sound collection device 13, and an operation device 14). The display device 15 displays an image instructed by the control device 11. The display device 15 is, for example, a liquid crystal display panel or an organic EL display panel.
[0061] The performance device 50 is an automatic musical instrument that functions not only as a playback device that executes automatic performance of the playback part of a musical piece, but also as an instrument that can be played manually by the user U1. Specifically, the performance device 50 includes a drive mechanism 21 and a sound generation mechanism 22, similar to the playback device 20 in each of the above-mentioned embodiments. Note that the performance device 50 of the fifth embodiment may be used as the playback device 20 of the first to fourth embodiments.
[0062] The manual performance by the user U1 is, for example, a performance by the user U1's physical movements such as pressing keys on a keyboard. The sound generation mechanism 22 operates in conjunction with the performance by the user U1, and musical tones are generated from the performance device 50. In parallel with the performance by the user U1, the performance device 50 sequentially outputs instruction data d representing instructions for the performance to the playback control system 10. The instruction data d specifies, for example, pitch and intensity, and specifies an action such as sound generation or muting. On the other hand, the user U2 plays the musical instrument 200. The musical instrument 200 is a natural musical instrument such as a stringed instrument that produces sound when played by the user U.
[0063] FIG. 13 is a block diagram illustrating a functional configuration of a playback control system 10 in the fifth embodiment. The control device 11 in the fifth embodiment executes a program stored in the storage device 12 to function as a preparation processing unit 35 in addition to the same elements as those in the first embodiment (performance analysis unit 31, playback control unit 32, and instruction receiving unit 33). The preparation processing unit 35 generates music data M (performance data D and reference data R) used in the playback control process Sa. Specifically, the preparation processing unit 35 generates music data M according to the performance of the performance device 50 by the user U1 and the performance of the musical instrument 200 by the user U2. The preparation processing unit 35 includes a first recording unit 41, a second recording unit 42, and a reference data generating unit 43.
[0064] The reference data generating unit 43 generates reference data R used in the playback control process Sa. Specifically, the reference data generating unit 43 executes a reference data generating process Sc (FIG. 14) before the start of the playback control process Sa to generate the reference data R. The reference data R generated by the reference data generating unit 43 is stored in the storage device 12.
[0065] During the period before the reference data generation process Sc is executed (hereinafter referred to as the "preparation period"), user U1 and user U2 play a piece of music together. Specifically, user U1 plays the playback part of the piece of music using the performance device 50, and user U2 plays the performance part of the piece of music using the musical instrument 200. The reference data generation process Sc is a process for generating reference data R using the results of users U1 and U2 playing the piece of music during the preparation period.
[0066] The first recording unit 41 acquires performance data D that specifies a plurality of sounds that the user U1 instructs the performance device 50 to play during the preparation period. Specifically, the first recording unit 41 generates performance data D in which instruction data d sequentially supplied from the performance device 50 according to the performance of the user U1 and time data specifying the interval between adjacent instruction data d are arranged in chronological order. The first recording unit 41 saves the performance data D in the storage device 12. The performance data D stored in the storage device 12 is used for the playback control process Sa, as exemplified in each of the above-mentioned embodiments. The performance data D acquired by the first recording unit 41 may be edited by the editing processing unit 34 exemplified in the fourth embodiment.
[0067] The second recording unit 42 acquires the sound signal Z (hereinafter referred to as "reference signal Zr1") generated by the sound collection device 13 during the preparation period. During the preparation period, in addition to the musical tones generated by the musical instrument 200 during the performance of the user U2, the musical tones generated by the performance device 50 during the performance of the user U1 reach the sound collection device 13. Therefore, the reference signal Zr1 is a signal representing a mixed sound of the musical tones of the musical instrument 200 and the musical tones of the performance device 50. As can be understood from the above explanation, the second recording unit 42 acquires the reference signal Zr1 representing the musical tones generated by the musical instrument 200 during the performance (first performance) of the musical piece using the musical instrument 200 and the musical tones generated by the performance device 50 during the performance of the musical instrument using the performance device 50. The second recording unit 42 stores the reference signal Zr1 in the storage device 12. As can be understood from the above explanation, during the preparation period, the performance data D and the reference signal Zr1 are stored in the storage device 12.
[0068] The reference data generating unit 43 generates the reference data R by a reference data generating process Sc using the reference signal Zr1 acquired by the second recording unit 42. In the above-mentioned embodiments, MIDI data in which instruction data and time data are arranged in a time series is exemplified as the reference data R. The reference data R in the fifth embodiment is data that specifies a performance period, a sounding time, and a pitch transition in a musical piece. The performance period is a period during which the user U2 plays the musical instrument 200 in the preparation period. The performance period can also be said as a period during which the musical instrument 200 sounds a musical tone. For example, the start and end times of the performance period are specified by the reference data R. The sounding time is a time when the sounding of each musical tone in the musical piece begins (i.e., onset) during the performance by the user U2 during the preparation period. For example, the time of the sounding time is specified by the reference data R. The pitch transition is a time series of the pitch of the musical tone sounded by the musical instrument 200 during the performance by the user U1 during the preparation period.
[0069] 14 is a flowchart illustrating a specific procedure of the reference data generation process Sc. After the performance data D and the reference signal Zr1 are acquired in the preparation period, the reference data generation process Sc is started, for example, in response to an instruction from the user U (U1, U2) via the operation device 14.
[0070] When the reference data generation process Sc is started, the reference data generation unit 43 generates a reference signal Zr2 by emphasizing the acoustic components of the musical tones produced by the musical instrument 200 in the reference signal Zr1 (Sc1). As described above, the reference signal Zr1 includes the acoustic components of the musical tones produced by the performance device 50 and the acoustic components of the musical tones produced by the musical instrument 200. The reference data generation unit 43 generates the reference signal Zr2 by suppressing the acoustic components of the musical tones of the performance device 50 from the reference signal Zr1.
[0071] For example, the reference data generating unit 43 generates the reference signal zr2 by subtracting the amplitude spectrogram of the musical sound of the performance device 50 from the amplitude spectrogram of the reference signal Zr1. The amplitude spectrogram of the musical sound of the performance device 50 is generated, for example, by a known sound source process that generates a musical sound signal representing the musical sound specified by the performance data D, and a frequency analysis such as a discrete Fourier transform of the musical sound signal. The degree to which the amplitude spectrogram of the musical sound of the performance device 50 is subtracted is adjusted according to an instruction from the user U via the operation device 14.
[0072] The process of generating the reference signal Zr2 from the reference signal Zr1 is not limited to the above example. For example, a known sound source separation technique may be used to emphasize the acoustic components of the musical sound of the musical instrument 200 in the reference signal Zr1. Also, for example, if the musical sound produced by the performance device 50 is unlikely to reach the sound collection device 13, step Sc1 may be omitted. In a form in which step Sc1 is omitted, the following processes are performed on the reference signal Zr1 instead of the reference signal Zr2. The reference signal Zr1 and the reference signal Zr2 are examples of the "first acoustic signal."
[0073] The control device 11 causes the display device 15 to display a preparation screen 60 of Fig. 15 in parallel with the reference data generation process Sc. A waveform 61 of the reference signal Zr2 is displayed on the preparation screen 60. The user U can adjust the range of the reference signal Zr2 that is displayed on the preparation screen 60 by operating the operation device 14.
[0074] As illustrated in Fig. 14, the reference data generating unit 43 analyzes the reference signal Zr2 to identify one or more performance periods in the reference signal Zr2 (Sc2). A first HMM (Hidden Markov Model) that estimates the performance period according to the intensity of the reference signal Zr2 is used to identify the performance period. In general, a period in which the signal intensity of the reference signal Zr2 exceeds a threshold value τ is identified as the performance period. Note that the method of identifying the performance period is not limited to the above example.
[0075] The preparation screen 60 displays the performance period identified from the reference signal Zr2. Specifically, as illustrated in FIG. 15, the control device 11 displays a portion 61a within the performance period and a portion 61b outside the performance period of the waveform 61 of the reference signal Zr2 in different display modes. The "display mode" refers to the characteristics of an image that can be visually distinguished by an observer. For example, the three attributes of color, that is, hue (tone), saturation, and brightness (gradation), as well as patterns or shapes, are also included in the concept of the "display mode." Note that the method of displaying the performance period is not limited to the above examples.
[0076] Furthermore, the preparation screen 60 displays an operation image 62 that the user U can operate using the operation device 14. The reference data generating unit 43 controls a threshold value τ applied to identifying the performance period in response to an instruction from the user U to the operation image 62. The smaller the threshold value τ set by the user U, the more likely it is that each point in the reference signal Zr2 will be determined to be a point in the performance period.
[0077] The reference data generating unit 43 identifies a plurality of sound generation time points in the reference signal Zr2 by analyzing the reference signal Zr2 (Sc3). As illustrated in Fig. 15, the preparation screen 60 displays an instruction image 63 representing the sound generation time points identified from the reference signal Zr2. The instruction image 63 is a vertical line arranged at each sound generation time point on the time axis. However, the specific form of the instruction image 63 is not limited to the above example.
[0078] The reference data generating unit 43 uses a second HMM consisting of a plurality of states corresponding to different pitches to identify the playing time point. The time point at which a transition occurs between different states (i.e., the time point at which the pitch changes) is identified as the sounding time point. In response to an instruction from the user U of the type of instrument 200, the reference data generating unit 43 may limit the range of state transitions in the second HMM (i.e., the range in which the pitch changes) to the range of the instrument 200. In other words, transitions to states corresponding to pitches outside the range of the instrument 200 are prohibited.
[0079] A first transition matrix Λ1 and a second transition matrix Λ2 are set for each state of the second HMM. The first transition matrix Λ1 and the second transition matrix Λ2 for each state are matrices that define the transition probability (self-transition probability) of the state itself and the transition probability of the state to another state. The numerical values of each transition probability are different between the first transition matrix Λ1 and the second transition matrix Λ2. Specifically, the transition probability to another state in the first transition matrix Λ1 is lower than the transition probability to another state in the second transition matrix Λ2. In other words, the self-transition probability in the first transition matrix Λ1 is higher than the self-transition probability in the second transition matrix Λ2. Therefore, when the first transition matrix Λ1 is applied to the second HMM, transitions between states are less likely to occur compared to when the second transition matrix Λ2 is applied to the second HMM. That is, the frequency with which each time point on the time axis is estimated as a sounding time point decreases.
[0080] The reference data generating unit 43 of the fifth embodiment applies a transition matrix Λ, which is calculated for each state by a weighted sum of a first transition matrix Λ1 and a second transition matrix Λ2, to each state of the second HMM. The transition matrix Λ is expressed, for example, by the following formula (1). The coefficient α in formula (1) is a numerical value in the range of 0 to 1. Λ=α Λ1+(1-α) Λ2 (1)
[0081] As illustrated in FIG. 15, the preparation screen 60 displays an operation image 64 that the user U can operate using the operation device 14. The reference data generating unit 43 controls the coefficient α applied to the specification of the performance time point in response to an instruction from the user U on the operation image 64. As can be understood from the formula (1), the larger the coefficient α, the greater the influence of the first transition matrix Λ1 on the transition matrix Λ, and the smaller the influence of the second transition matrix Λ2 on the transition matrix Λ. Therefore, transitions between states in the second HMM are less likely to occur. That is, the probability that each time point of the reference signal Zr2 is determined to be a sounding time point decreases. As can be understood from the above explanation, there is a tendency that the larger the coefficient α, the smaller the number of sounding times specified for the reference signal Zr2. In other words, the smaller the coefficient α, the greater the number of sounding times. The user U operates the operation image 64 while checking the preparation screen 60, thereby adjusting the coefficient α so that the number of sounding times displayed by the instruction image 63 is an appropriate number. Note that the method of specifying the sounding time point is not limited to the above example.
[0082] The reference data generating unit 43 identifies pitch transitions by analyzing the reference signal Zr2 (Sc4). For example, an estimation model that learns the relationship between the frequency characteristics of the reference signal Zr2 and pitch transitions by machine learning is used to identify the pitch transitions. The estimation model is, for example, a deep neural network such as a convolutional neural network or a recurrent neural network. For example, control data including a spectrum generated by constant Q conversion of the reference signal Zr2 is supplied to the estimation model. The estimation model outputs a time series of pitches (i.e., pitch transitions) that correlate with the control data. Note that the method of identifying pitch transitions is not limited to the above examples. In addition, the order of emphasizing the acoustic components of the musical tones (Sc1), identifying the performance period (Sc2), identifying the sounding time point (Sc3), and identifying the pitch transitions (Sc4) may be changed arbitrarily. For example, the acoustic components of the musical tones of the performance device 50 may be suppressed from the reference signal Zr1 by using a numerical value (e.g., the posterior probability of the pitch) calculated in identifying the pitch transitions (Sc4).
[0083] The reference data generating unit 43 judges whether or not an instruction has been received from the user U (Sc5). Specifically, it judges whether or not the user U has instructed to change the threshold value τ or the coefficient α. If an instruction from the user U has been received (Sc5: YES), the reference data generating unit 43 identifies the performance period, the sounding time point, and the pitch transition by processing (Sc1 to Sc4) that applies the changed threshold value τ or coefficient α.
[0084] It is assumed that the processing load for identifying the sounding time exceeds the processing load for identifying the performance period. Considering the above circumstances, a configuration is assumed in which the identification of the sounding time is performed less frequently than the identification of the performance period. For example, the reference data generating unit 43 repeats the identification of the performance period (Sc2) during the process in which the user U changes the threshold value τ by operating the operation image 62. For example, the identification of the performance period (Sc2) is repeated during the period in which the user U drags the operation image 62 to change the threshold value τ. On the other hand, the reference data generating unit 43 does not perform the identification of the sounding time during the process in which the user U changes the coefficient α by operating the operation image 64, but performs the identification of the sounding time (Sc3) when the change of the coefficient α is completed (i.e., the coefficient α after the change is confirmed). For example, the sounding time is identified at the stage when the user U finishes the operation of dragging the operation image 62 to change the coefficient α. The above configuration has the advantage that the performance period changes quickly in response to the instruction of the user U, while the processing load required for identifying the sounding time is reduced.
[0085] If no instruction from the user U is accepted (Sc5: NO), the reference data generating unit 43 judges whether or not an instruction to save the reference data R has been accepted from the user U (Sc6). If no instruction is accepted (Sc6: NO), the reference data generating unit 43 transitions to step Sc5. On the other hand, if an instruction to save is accepted from the user U (Sc6: YES), the reference data generating unit 43 saves the reference data R, which specifies the current performance period, sounding time, and pitch transition, in the storage device 12 (Sc7).
[0086] The reference data R generated by the reference data generation process Sc exemplified above is applied to the playback control process Sa. The specific procedure of the playback control process Sa in the fifth embodiment is the same as that of each of the above-mentioned embodiments. For example, the performance analysis unit 31 estimates the playing position X in parallel with the performance by collating the audio signal Z representing the musical tone generated by the musical instrument 200 in the performance of the performance part by the user U2 (second performance) with the reference data R generated by the reference data generation process Sc. In addition, the playback control unit 32 causes the performance device 50 to play each note of the playback part designated by the performance data D acquired by the first recording unit 41. Specifically, the playback control unit 32 causes the playback of each note of the playback part to follow the performance by the user U2 according to the result of the estimation by the performance analysis unit 31.
[0087] In the above description, a single performance device 50 is used to obtain the performance data D during the preparation period and for automatic performance by the playback control process Sa. However, a playback device 20 that executes automatic performance by the playback control process Sa may be used separately from the performance device 50 used to obtain the performance data D during the preparation period. In other words, the automatic performance function is not essential for the performance device 50. Also, different musical instruments 20 may be used to obtain the reference signal Zr1 during the preparation period and for performance in parallel with the playback control process Sa.
[0088] Also, similar to the above-mentioned embodiments, the playback control unit 32 continues the playback of the note N1 being played back at the time when the user U2 issues the first instruction Q1, even after the first instruction Q1 is issued, until the end point of the note N1 specified by the performance data D. Also, after the playback of the note N1 is stopped, the playback control unit 32 starts the playback of the note N2 immediately following the note N1, triggered by the second instruction Q2 by the user U. Therefore, the fifth embodiment also achieves the same effects as the above-mentioned embodiments.
[0089] Moreover, according to the fifth embodiment, the reference data R is generated according to the result of the user U2 playing the musical instrument 200. Therefore, for any piece of music for which reference data R is not prepared, the reference data R reflecting the performance of the user U2 can be generated.
[0090] The control device 11 of the fifth embodiment causes the display device 15 to display a playback screen 70 of Fig. 16 in parallel with the playback control process Sa. The playback image 70 includes an operation image 71, a playback image 72, an instruction image 73, an instruction image 74, and an operation image 75.
[0091] The operation image 71 is an image that allows the user U to instruct the performance device 50 to start automatic performance (start of the playback control process Sa). The playback image 72 is an image that represents the current playback position Y of the performance device 50. Specifically, the playback image 72 is made up of a time axis 721 that represents the entire section of the music piece, and an instruction image 722 that represents the current playback position Y. The instruction image 722 moves along the time axis 721 as the automatic performance progresses.
[0092] The instruction image 73 is an image for notifying the user U of the sound generation time points specified by the reference data R. Specifically, the control device 11 changes the display mode of the instruction image 73 when the playback position Y reaches each sound generation time point specified by the reference data R. For example, the instruction image 73 is momentarily enlarged when the playback position Y coincides with the sound generation time point. The user U can check each sound generation time point specified by the reference data R by visually checking the instruction image 73. Therefore, for the performance of the user U2 in the preparation period, the excess or deficiency of the sound generation time points estimated by the reference data generation process Sc (i.e., the appropriateness of the process of identifying the sound generation time points) can be visually confirmed.
[0093] The indication image 74 is an image for notifying the user U whether or not the volume of the musical sound produced by the musical instrument 200 is appropriate for estimating the performance position X with high accuracy. When the volume σ of the audio signal Z is too high or too low, there is a tendency for the accuracy of estimation of the performance position X by the performance analysis unit 31 to decrease. Taking the above tendency into consideration, the control device 11 changes the display mode of the indication image 74 according to the volume σ of the audio signal Z.
[0094] Specifically, the control device 11 maintains the display mode of the instruction image 74 in the first mode when the volume σ of the audio signal Z is equal to or greater than a threshold σL and equal to or less than a threshold σH. The threshold σH is a numerical value greater than the threshold σL. Each of the thresholds σL and σH is set experimentally or statistically so that the performance position X is estimated with a target accuracy when the volume σ of the audio signal Z is a numerical value within a range of equal to or greater than the threshold σL and equal to or less than the threshold σH.
[0095] Moreover, when the volume σ of the audio signal Z is below the threshold σL, the control device 11 changes the display mode of the instruction image 74 to a second mode different from the first mode. That is, when the volume σ is too small for estimating the playing position X with high accuracy, the instruction image 74 is displayed in the second mode. On the other hand, when the volume σ of the audio signal Z is above the threshold σH, the control device 11 changes the display mode of the instruction image 74 to a third mode different from the first mode. That is, when the volume σ is too large for estimating the playing position X with high accuracy, the instruction image 74 is displayed in the third mode. As can be understood from the above explanation, the decrease in the estimation accuracy of the playing position X is notified to the user U by the change in the display mode of the instruction image 74. Note that there is no difference between the second mode and the third mode.
[0096] In the above description, attention has been focused on the volume σ of the audio signal Z, but the index serving as a reference for changing the display mode of the instruction image 74 is not limited to the volume σ. For example, the control device 11 may control the display mode of the instruction image 74 according to the likelihood of the pitch estimated in step Sc4 of the reference data generation process Sc.
[0097] 16 is an image that allows the user U to adjust the performance speed of the automatic performance. By operating the operation image 75, the user U can specify the performance speed of the automatic performance by multiplying it by a predetermined reference value (i.e., a relative value). The playback control unit 32 controls the automatic performance by the performance device 50 so that the playback position Y follows the performance position X, using the numerical value specified by the user U as the reference value for the performance speed.
[0098] F: Variation Specific modified embodiments added to each of the above-mentioned embodiments are exemplified below. Two or more embodiments selected from the following examples may be appropriately combined as long as they are not mutually contradictory.
[0099] (1) In each of the above-mentioned embodiments, the instruction receiving unit 33 receives an operation for transitioning the operation device 14 from a released state to an operating state as the first instruction Q1, but the form of the first instruction Q1 is not limited to the above examples. For example, a specific motion by the user U is detected as the first instruction Q1. To detect the motion of the user U, various detection devices such as an imaging device or an acceleration sensor are used. For example, the instruction receiving unit 33 identifies various motions such as the motion of the user U lifting one hand, the motion of lifting the musical instrument 200, or the motion of breathing (for example, the motion of inhaling) as the first instruction Q1. The breathing by the user U is, for example, a breath when playing a wind instrument as the musical instrument 200. The operation speed V1 in the second embodiment is comprehensively expressed as the speed of the motion of the user U identified as the first instruction Q1.
[0100] Specific data (hereinafter referred to as "first data") meaning a first instruction Q1 may be included in the performance data D. The first data is, for example, data meaning a fermata in a piece of music. The instruction receiving unit 33 determines that the first instruction Q1 has occurred when the playback position Y reaches the point of the first data. As can be understood from the above explanation, the first instruction Q1 is not limited to an instruction from the user U. Note that, when the dispersion degree Δ exceeds the threshold value Δth in the editing process Sb, the editing processing unit 34 may add the first data to the note N1.
[0101] (2) In each of the above-described embodiments, the instruction receiving unit 33 receives an operation for transitioning the operating device 14 from an operating state to a released state as the second instruction Q2, but the aspect of the second instruction Q2 is not limited to the above examples. For example, similar to the first instruction Q1 in the first embodiment, the instruction receiving unit 33 may receive an operation for transitioning the operating device 14 from a released state to an operating state as the second instruction Q2. In other words, two operations including stepping on and releasing the movable part 141 may be detected as the first instruction Q1 and the second instruction Q2.
[0102] Also, a specific motion of the user U may be detected as the second instruction Q2. To detect the motion of the user U, various detection devices such as an imaging device or an acceleration sensor are used. For example, the instruction receiving unit 33 identifies various motions such as the motion of the user U lowering one hand, the motion of lowering the musical instrument 200, or the motion of breathing (for example, the motion of exhaling) as the second instruction Q2. The breathing by the user U is, for example, a breath when playing a wind instrument as the musical instrument 200. The operation speed V2 in the second embodiment is comprehensively expressed as the speed of the motion of the user U identified as the second instruction Q2.
[0103] Specific data (hereinafter referred to as "second data") representing the second instruction Q2 may be included in the performance data D. The second data is, for example, data representing a fermata in a piece of music. The instruction receiving unit 33 determines that the second instruction Q2 has occurred when the playback position Y reaches the point of the second data. As can be understood from the above explanation, the second instruction Q2 is not limited to an instruction from the user U.
[0104] As shown in the above example, a configuration is assumed in which one of a series of mutually paired actions of the user U is accepted as the first instruction Q1, and the other is accepted as the second instruction Q2. For example, the action of the user U raising one hand is accepted as the first instruction Q1, and the action of lowering the one hand following that action is accepted as the second instruction Q2. Also, the action of the user U lifting the musical instrument 200 is accepted as the first instruction Q1, and the action of lowering the musical instrument 200 following that action is accepted as the second instruction Q2. Similarly, the action of the user U inhaling is accepted as the first instruction Q1, and the action of exhaling following that action is accepted as the second instruction Q2.
[0105] However, the first instruction Q1 and the second instruction Q2 do not have to be the same type of action by the user U. That is, separate actions that the user U can perform independently of each other may be detected as the first instruction Q1 and the second instruction Q2. For example, the instruction receiving unit 33 may detect an operation on the operating device 14 as the first instruction Q1, and detect another action, such as an action of lifting the musical instrument 200 or an action of breathing, as the second instruction Q2.
[0106] (3) In each of the above-mentioned embodiments, an automatic musical instrument is exemplified as the playback device 20, but the configuration of the playback device 20 is not limited to the above examples. For example, a sound source system including a sound source device that generates an audio signal of a musical tone in response to an instruction from the playback control system 10 and a sound emitting device that reproduces the musical tone represented by the audio signal may be adopted as the playback device 20. The sound source device is realized as a hardware sound source or a software sound source. The same applies to the performance device 50 in the fifth embodiment.
[0107] (4) In each of the above-mentioned embodiments, the playback device 20 is controlled using performance data D that specifies the time series of multiple notes of a piece of music, but the format of data used to control the playback device 20 is not limited to the above examples. For example, waveform data that represents the waveforms of multiple sounds that make up the playback part may be used to control the playback device 20. The waveform data is composed of a time series of multiple samples. The playback control unit 32 supplies the playback device 20 with a sample from the waveform data that corresponds to the playback position Y. The playback device 20 is a sound emission system that reproduces the sound represented by the time series of samples supplied from the playback control system 10.
[0108] The waveform data is added with sound generation point data representing the start point of each of the multiple sounds (hereinafter referred to as "sound generation point"). When the instruction receiving unit 33 receives the first instruction Q1, the playback control unit 32 causes the playback device 20 to play the sound represented by the waveform data until the time immediately before the sound generation point located immediately after the first instruction Q1, and waits for the second instruction Q2 after the playback. That is, the playback of the sound (first sound) being played at the time of generation of the first instruction Q1 is continued until the end point of the first sound. On the other hand, when the instruction receiving unit 33 receives the second instruction Q2, the playback control unit 32 causes the playback device 20 to play the sound (second sound) represented by the waveform data from the sound generation point immediately after the section that has already been played (the sound generation point located immediately after the first instruction Q1). That is, after the playback of the first sound is stopped, the playback of the second sound immediately after the first sound is started in response to the second instruction Q2. In the above explanation, modified examples of the first to fourth embodiments equipped with the playback device 20 have been given, but the format of the data for controlling the automatic performance by the performance device 50 of the fifth embodiment is similarly not limited to the performance data D, and for example, the waveform data exemplified above may be used.
[0109] As can be understood from the above description, the performance data D exemplified in each of the above-mentioned embodiments and the waveform data exemplified in the modified examples are collectively expressed as sound data representing a time series of a plurality of sounds.
[0110] (5) The playback control system 10 may be realized by a server device that communicates with a terminal device such as a smartphone or a tablet terminal. The terminal device includes a sound collection device 13 that generates an audio signal Z according to a performance by a user U, and a playback device 20 (or a performance device 50 in the fifth embodiment) that plays a piece of music according to an instruction from the playback control system 10. The terminal device transmits the audio signal Z generated by the sound collection device 13 and a first instruction Q1 and a second instruction Q2 according to the action of the user U to the playback control system 10 via a communication network. The playback control system 10 causes the playback device 20 of the terminal device to play the playback part of the music according to the playing position X estimated from the audio signal Z and the first instruction Q1 and the second instruction Q2 received from the terminal device. The performance analysis unit 31 may be mounted on the terminal device. The terminal device transmits the playing position X estimated by the performance analysis unit 31 to the playback control system 10. In the above configuration, the performance analysis unit 31 is omitted from the playback control system 10.
[0111] (6) The functions of the playback control system 10 exemplified above are realized by the cooperation of one or more processors constituting the control device 11 and the program stored in the storage device 12, as described above. The program according to the present disclosure can be provided in a form stored in a computer-readable recording medium and installed in a computer. The recording medium is, for example, a non-transitory recording medium, and a good example is an optical recording medium (optical disk) such as a CD-ROM, but also includes any known type of recording medium such as a semiconductor recording medium or a magnetic recording medium. Note that the non-transitory recording medium includes any recording medium except for a transient, propagating signal, and does not exclude volatile recording media. In addition, in a configuration in which a distribution device distributes a program via a communication network, a storage device that stores the program in the distribution device corresponds to the non-transitory recording medium described above.
[0112] G: Notes From the above-described exemplary embodiments, the following configurations can be understood, for example.
[0113] A playback control method according to one aspect (aspect 1) of the present disclosure is a method for playing back a plurality of sounds using sound data representing a time series of the sounds, which plays back a first sound corresponding to a first instruction among the plurality of sounds, and after the playback of the first sound is stopped, starts playback of a second sound immediately after the first sound among the plurality of sounds in response to a second instruction from a user. According to the above aspect, after the playback of the first sound corresponding to the first instruction is stopped, starts playback of the second sound immediately after the first sound in response to a second instruction from the user. Therefore, the interval between the first sound and the second sound (e.g., the duration of the rest period) can be appropriately controlled according to the time points of the first instruction and the second instruction.
[0114] "Sound data" is data in any format that represents a time series of multiple sounds. For example, performance data that specifies the sounding period (specifically, sounding and muting) for each of multiple notes, or waveform data that represents the waveforms of multiple sounds on a time axis, are included in the concept of "sound data."
[0115] The "first instruction" is, for example, an instruction corresponding to an action by a user, or an instruction added to the sound data. An instruction corresponding to an action by a user is, for example, an instruction given by the user operating an operating device. The first instruction may also be generated when the user performs a specific action (for example, lifting an instrument or taking a breath while playing a wind instrument). An instruction added to the sound data is, for example, various performance instructions such as a fermata, which means the extension of a note or rest.
[0116] The "second instruction" is, for example, an instruction corresponding to an action by a user. An instruction corresponding to an action by a user is, for example, an instruction given by the user operating an operating device. In addition, the second instruction may be generated when the user performs a specific action (for example, lifting an instrument or taking a breath while playing a wind instrument).
[0117] "Starting playback of the second sound in response to the second instruction" means that playback of the second sound is started on the condition that the second instruction is received, and the relationship between the time of the second instruction and the time at which playback of the second sound is started is not important. For example, in addition to the mode in which playback of the second sound is started at or immediately after the second instruction, the mode in which playback of the second sound is started when a predetermined time has elapsed since the second instruction is also included in the mode in which "playback of the second sound is started in response to the second instruction."
[0118] The "reproduction" of multiple sounds means emitting each sound as a sound wave. Therefore, for example, the concept of "reproduction" includes automatic performance in which an automatic musical instrument such as an automatic piano reproduces multiple sounds, or the emission of multiple sounds by a sound source device and a sound emission device.
[0119] In a specific example (aspect 2) of aspect 1, the first sound is one of the multiple sounds being played at the time when a first instruction is issued, and in playing the first sound, the playback of the first sound is continued after the first instruction is issued until the end of the first sound represented by the sound data. In the above aspect, even after the first instruction is issued, the playback of the first sound is continued until the end of the first sound represented by the sound data. Therefore, compared to a configuration in which the playback of the first sound is stopped at the time of the first instruction, it is possible to appropriately continue the first sound according to the sound data.
[0120] In a specific example (Aspect 3) of Aspect 1, the plurality of sounds constitute a piece of music, a point in time in the piece of music that the user is playing is estimated in parallel with the playback of the plurality of sounds, and in the playback of the plurality of sounds, the playback of the plurality of sounds is made to follow the performance of the piece of music according to the result of the estimation. In the above aspect, since the playback of the plurality of sounds follows the performance of the piece of music by the user, it is possible to appropriately reflect the user's intention (e.g., performance expression) or preference in the playback of the plurality of sounds.
[0121] In a specific example (Aspect 4) of any one of Aspects 1 to 3, the sound data is performance data that specifies a sound generation period for each of the plurality of sounds, and in playing the first sound, after the first instruction is generated, the playback of the first sound is continued until the end of the sound generation period specified for the first sound by the performance data. In the above aspects, even after the first instruction is generated, the playback of the first sound is continued until the end of the sound generation period specified for the first sound by the performance data. Therefore, compared to a configuration in which the playback of the first sound is stopped at the time of the first instruction, it is possible to appropriately continue the first sound over the sound generation period specified by the performance data.
[0122] In a specific example (Aspect 5) of any one of Aspects 1 to 4, the first instruction and the second instruction are generated in response to an operation of an operating device by the user. According to the above aspect, the user can change the interval between the first sound and the second sound to an appropriate time length according to the user's intention or preference by operating the operating device.
[0123] In a specific example (aspect 6) of aspect 5, the first instruction is generated by the user's operation of transitioning the operating device from a first state to a second state, and the second instruction is generated by the user's operation of transitioning the operating device from the second state to the first state. According to the above aspect, the first instruction is generated by the user transitioning the operating device from the first state to the second state, and after the second state is maintained, the second instruction is generated by the user transitioning the operating device from the second state to the first state at a time point desired by the user. Therefore, the operation of the operating device by the user is simplified compared to a configuration in which an operation of transitioning the operating device from the first state to the second state is required for each of the first instruction and the second instruction.
[0124] In a specific example (aspect 7) of aspect 5 or aspect 6, the duration of the first sound is controlled according to the speed of operation of the operation device. According to the above aspect, the user can adjust the duration of the first sound according to the speed of operation of the operation device. In addition, since the operation device for giving the first instruction and the second instruction is also used to adjust the duration of the first sound, there is an advantage that the operation by the user is simplified compared to a configuration in which the user operates separate devices for each of giving the first instruction and the second instruction and adjusting the duration of the first sound.
[0125] In a specific example (Aspect 8) of any of Aspects 5 to 7, the time point at which playback of the second sound starts is controlled according to the speed of operation of the operation device. According to the above aspects, the user can adjust the start point of playback of the second sound according to the speed of operation of the operation device. In addition, since the operation device for giving the first instruction and the second instruction is also used to adjust the start point of the second sound, there is an advantage that the operation by the user is simplified compared to a configuration in which the user operates separate devices for each of giving the first instruction and the second instruction and adjusting the start point of the second sound.
[0126] A playback control method according to one aspect (aspect 9) of the present disclosure is a method for playing a plurality of sounds using sound data representing a time series of the sounds, and continues a first sound among the plurality of sounds that is being played at the time when a first instruction is issued until the end point of the first sound represented by the sound data. In the above aspect, even after the first instruction is issued, the playback of the first sound continues until the end point of the first sound represented by the sound data. Therefore, compared to a configuration in which the playback of the first sound is stopped at the time of the first instruction, it is possible to appropriately continue the first sound according to the sound data.
[0127] A playback control system according to one aspect (aspect 10) of the present disclosure includes a playback control unit that plays back a plurality of sounds using sound data representing a time series of the sounds, and the playback control unit plays back a first sound corresponding to a first instruction among the plurality of sounds, and after the playback of the first sound is stopped, starts playing a second sound immediately after the first sound among the plurality of sounds in response to a second instruction from a user. According to the above aspect, after the playback of the first sound corresponding to the first instruction is stopped, the playback of the second sound immediately after the first sound is started in response to a second instruction from the user. Therefore, the interval between the first sound and the second sound (e.g., the duration of the rest period) can be appropriately controlled according to the time points of the first instruction and the second instruction.
[0128] A program according to one aspect (aspect 11) of the present disclosure causes a computer to function as a playback control unit that uses sound data representing a time series of a plurality of sounds to play back the plurality of sounds, the playback control unit playing back a first sound of the plurality of sounds corresponding to a first instruction, and after the playback of the first sound is stopped, starting playback of a second sound of the plurality of sounds immediately after the first sound in response to a second instruction from a user. According to the above aspect, after the playback of the first sound corresponding to the first instruction is stopped, the playback of the second sound immediately after the first sound is started in response to a second instruction from the user. Therefore, the interval between the first sound and the second sound (e.g., the duration of the rest period) can be appropriately controlled according to the time points of the first instruction and the second instruction.
[0129] In order to estimate the time (performance position) in a piece of music that a user is playing, reference data that can be compared with the performance by the user is required. However, there may be a situation where reference data for a desired piece of music is not available.
[0130] In consideration of the above circumstances, a playback control system according to one aspect (aspect 12) of the present disclosure includes a first recording unit that acquires performance data specifying a plurality of sounds instructed to a playback device, a second recording unit that acquires a first audio signal representing a musical tone generated by a musical instrument in a first performance of the musical piece using the musical instrument, a reference data generating unit that generates reference data representing at least a sound time of each musical tone in the first performance from the first audio signal, a performance analysis unit that estimates a time of the musical tone being played in the musical piece in parallel with the second performance by comparing a second audio signal representing a musical tone generated by the musical instrument in a second performance of the musical piece using the musical instrument with the reference data, and a playback control unit that reproduces the plurality of sounds specified by the performance data and causes the reproduction of the plurality of sounds to follow the second performance according to a result of the estimation by the performance analysis unit. According to the above configuration, reference data representing a sound time of each musical tone in the performance is generated from a first audio signal representing a musical tone generated by the musical instrument in a performance of the musical piece using the musical instrument. Therefore, it is not necessary to prepare reference data for a desired musical piece in advance.
[0131] In a specific example (aspect 13) of aspect 12, the playback control unit continues a first sound of the multiple sounds that is being played at the time when a first instruction is issued until the end point of the first sound represented by the sound data, and after the playback of the first sound is stopped, starts playback of a second sound of the multiple sounds immediately after the first sound in response to a second instruction from the user. According to the above aspect, after the playback of the first sound corresponding to the first instruction is stopped, playback of the second sound immediately after the first sound is started in response to a second instruction from the user. Therefore, the interval between the first sound and the second sound (e.g., the duration of a rest period) can be appropriately controlled according to the time points of the first instruction and the second instruction. [Explanation of symbols]
[0132] 100...playback system, 200...musical instrument, 10...playback control system, 11...control device, 12...storage device, 13...sound collection device, 14...operation device, 141...movable part, 20...playback device, 21...driving mechanism, 22...sound generation mechanism, 50...performance device, 31...performance analysis unit, 32...playback control unit, 33...instruction receiving unit, 34...editing processing unit, 41...first recording unit, 42...second recording unit, 43...reference data generation unit.
Claims
1. A method for reproducing a plurality of sounds by using sound data representing a time series of the plurality of sounds constituting a piece of music, comprising: estimating a point in time within the piece of music that a user is playing in parallel with the playback of the plurality of sounds; According to the result of the estimation, the reproduction of the plurality of sounds is synchronized with the performance of the musical piece; In the reproduction of the plurality of sounds, playing a first sound corresponding to a first instruction among the plurality of sounds; After the reproduction of the first sound is stopped, reproduction of a second sound immediately after the first sound among the plurality of sounds is started in response to a second instruction from a user. A computer-implemented playback control method.
2. A method for reproducing a plurality of sounds by using sound data representing a time series of the sounds, comprising: continuing a first sound among the plurality of sounds that is being reproduced at the time when a first instruction is issued until an end point of the first sound represented by the sound data; after stopping the reproduction of the first sound, starting reproduction of a second sound immediately after the first sound among the plurality of sounds in response to a second instruction from a user; the first instruction and the second instruction are generated in response to an operation of an operation device by the user, The duration of the first sound is controlled in accordance with the speed of the operation of the operating device. A computer-implemented playback control method.
3. A method for reproducing a plurality of sounds by using sound data representing a time series of the sounds, comprising: continuing a first sound among the plurality of sounds that is being reproduced at the time when a first instruction is issued until an end point of the first sound represented by the sound data; after stopping the reproduction of the first sound, starting reproduction of a second sound immediately after the first sound among the plurality of sounds in response to a second instruction from a user; the first instruction and the second instruction are generated in response to an operation of an operation device by the user, A time point at which the reproduction of the second sound is started is controlled according to a speed of the operation of the operation device. A computer-implemented playback control method.
4. the first instruction is generated by an operation by the user to transition the operation device from a first state to a second state, The second instruction is generated by an operation by the user to transition the operation device from the second state to the first state. The playback control method according to claim 2 or 3.
5. the sound data is performance data that specifies a sound generation period for each of the plurality of sounds; In the reproduction of the first sound, after the first instruction is generated, the reproduction of the first sound is continued until the end point of the sound generation period designated by the performance data for the first sound.
5. The playback control method according to claim 1.
6. a first recording unit for acquiring performance data specifying a plurality of sounds instructed to a performance device; a second recording unit that acquires a first audio signal representing a musical tone generated by a musical instrument during a first performance of a piece of music using the musical instrument; a reference data generating unit that generates reference data representing at least a sounding time point of each musical tone generated by the first performance from the first audio signal; a performance analysis unit that estimates a time point in the musical piece being played in parallel with the second performance by comparing a second audio signal representing a musical tone produced by the musical instrument in a second performance of the musical piece using the musical instrument with the reference data; a playback control unit that plays back a plurality of sounds designated by the performance data, the playback control unit causing the playback of the plurality of sounds to follow the second performance in accordance with a result of estimation by the performance analysis unit; Equipped with The playback control unit a first sound among the plurality of sounds that is being reproduced at the time when a first instruction is issued is continued until an end point of the first sound represented by the performance data; After the reproduction of the first sound is stopped, reproduction of a second sound immediately after the first sound among the plurality of sounds is started in response to a second instruction from a user. Playback control system.
Citation Information
Patent Citations
Wave data analysis method, wave data analyzer, and recording medium
JP2002215163A
Automatic player and method for setting tempo of automatic playing
JP2003091279A
Music reproducing device of portable terminal and music reproducing program used by portable terminal
JP2003131666A
Accompaniment data generation device and program
JP2012203216A
Information providing device
JP2016099512A
