Play control method and play control system

Abstract

In the playing of a plurality of sounds, the interval of two sounds before and after is appropriately controlled. A playing control system (10) has a playing control section (32) which causes a playing device (20) to play a plurality of notes using performance data (D) representing a time series of the plurality of notes. The playing control section (32) causes a first sound being played at the time of generation of a first instruction (Q1) among the plurality of notes to continue until the end of the first sound represented by the performance data (D), and after the playing of the first sound stops, starts the playing of a second sound following the first sound among the plurality of notes as a trigger from a second instruction (Q2) from a user.

Description

Technical Field

[0001] This invention relates to a technique for controlling the playback of sound. Background Technology

[0002] Techniques have been proposed to make the playback of music follow the performance of the performer. For example, Patent Document 1 discloses a technique that analyzes the musical tones produced by the performance of a piece of music, thereby estimating the position of the performance within the music, and controlling the automatic performance of the music accordingly based on the estimation result.

[0003] Patent Document 1: Japanese Patent Application Publication No. 2017-207615

[0004] Sometimes, in response to the performer's intention (e.g., musical expression) or performance markings such as Feuermann, the performer pauses during the performance of a piece. The length of the pause varies depending on the performer's intention or preference, and can differ for each performer or for each performance by the performer. Therefore, it is difficult to appropriately control the interval between two notes before and after a pause in accordance with the performer's intention or preference during the playback of a piece. Considering the above, one of the objectives of one aspect of the present invention is to appropriately control the interval between two notes during the playback of multiple notes. Summary of the Invention

[0005] To address the above-mentioned issues, one aspect of the present invention relates to a playback control method that uses sound data representing a time sequence of multiple sounds to play the multiple sounds. In this playback control method, the first sound that is playing at the moment a first instruction is generated continues until the end of the first sound represented by the sound data. After the playback of the first sound stops, the playback of the second sound that immediately follows the first sound begins, triggered by a second instruction from the user.

[0006] Other aspects of the present invention involve a playback control method that uses sound data representing a time sequence of multiple notes constituting a musical piece to play the multiple notes. In this playback control method, the moment when the user is playing the music is estimated in parallel with the playback of the multiple notes. Corresponding to the result of the estimation, the playback of the multiple notes follows the performance of the music. During the playback of the multiple notes, the first note among the multiple notes corresponding to the first instruction is played. After the playback of the first note stops, the playback of the second note among the multiple notes that immediately follows the first note is started based on the second instruction from the user.

[0007] Other aspects of the present invention involve a playback control method that uses sound data representing a time sequence of multiple sounds to play the multiple sounds, such that the first sound that is being played at the moment the first indication is generated continues until the end of the first sound represented by the sound data. Attached Figure Description

[0008] Figure 1 This is a block diagram illustrating the structure of the playback system according to the first embodiment.

[0009] Figure 2 This is a diagram of the music data.

[0010] Figure 3 It is an explanatory diagram of the structure and status of the operating device.

[0011] Figure 4 This is a block diagram illustrating the functional structure of a playback control system.

[0012] Figure 5 This is an explanatory diagram illustrating the relationship between the playback of the playback part via the playback device and the first and second instructions.

[0013] Figure 6 This is a flowchart illustrating the specific sequence of playback control processing.

[0014] Figure 7 This is an explanatory diagram relating to the state of the operating device in the second embodiment.

[0015] Figure 8 This is an explanatory diagram relating to the state of the operating device in the third embodiment.

[0016] Figure 9 This is a block diagram illustrating the functional structure of the playback control system in the fourth embodiment.

[0017] Figure 10 This is an explanatory diagram of the editing and processing department's operations.

[0018] Figure 11 It is a flowchart illustrating the specific sequence of editing processes.

[0019] Figure 12 This is a block diagram illustrating the structure of the playback system in the fifth embodiment.

[0020] Figure 13 This is a block diagram illustrating the functional structure of the playback control system in the fifth embodiment.

[0021] Figure 14 This is a flowchart illustrating the specific sequence of reference data generation and processing.

[0022] Figure 15This is a schematic diagram of a preparation screen that is displayed in parallel with the reference data generation and processing.

[0023] Figure 16 This is a schematic diagram of the playback screen displayed in parallel with the playback control processing. Detailed Implementation

[0024] A: Implementation Method 1

[0025] Figure 1 This is a block diagram illustrating the structure of the playback system 100 according to the first embodiment. The playback system 100 is provided in the acoustic space where the user U is located. The user U is, for example, a performer who plays a specific part of a piece of music (hereinafter referred to as "playing part") using an instrument 200 such as a stringed instrument. The playback system 100 is a computer system that plays the music in parallel with the playing part performed by the user U. Specifically, the playback system 100 plays the parts other than the playing part (hereinafter referred to as "playing part") among the multiple parts constituting the music. The playing part is, for example, the part constituting the main melody of the music. The playing part is, for example, one or more parts constituting the accompaniment of the music. As understood from the above description, the playing of the playing part by the user U and the playback of the playing part by the playback system 100 are performed in parallel, thereby realizing the performance of the music. In addition, the playing part and the playing part may also be common parts of the music.

[0026] The playback system 100 includes a playback control system 10 and a playback device 20. The playback control system 10 and the playback device 20 are separately configured and communicate with each other via wired or wireless means. Alternatively, the playback control system 10 and the playback device 20 can be configured as a single unit.

[0027] The playback device 20 plays the playback part of the music based on control performed by the playback control system 10. Specifically, the playback device 20 is an automatically playing instrument that performs the automatic playing of the playback part. For example, an automatically playing instrument of a different type than the instrument 200 played by the user U (e.g., an automatically playing piano) is used as the playback device 20. As understood from the above description, automatic playing is a mode of "playing".

[0028] The playback device 20 of the first embodiment includes a drive mechanism 21 and a sound-producing mechanism 22. The sound-producing mechanism 22 is a mechanism that emits musical sounds. Specifically, the sound-producing mechanism 22, similar to a keyboard instrument in a natural musical instrument, has a striking mechanism for each key, which is linked to the displacement of each key on the keyboard to make the string (sound source) sound. The drive mechanism 21 performs automatic performance of music by driving the sound-producing mechanism 22. The sound-producing mechanism 22 is driven by the drive mechanism 21 in response to instructions from the playback control system 10, thereby realizing the automatic performance of the played parts.

[0029] The playback control system 10 is a computer system that controls the playback of the sound output executed by the playback device 20, and includes a control device 11, a storage device 12, a pickup device 13, and an operation device 14. The playback control system 10 can be implemented, for example, through a mobile terminal device such as a smartphone or tablet, or a mobile or fixed terminal device such as a personal computer. Furthermore, the playback control system 10 can be implemented not only as a single device, but also as multiple devices configured as separate units.

[0030] The control device 11 is a single or multiple processors that control the various elements of the playback control system 10. Specifically, the control device 11 is composed of one or more processors such as CPU (Central Processing Unit), SPU (Sound Processing Unit), DSP (Digital Signal Processor), FPGA (Field Programmable Gate Array), or ASIC (Application Specific Integrated Circuit).

[0031] Storage device 12 is a single or multiple memory devices that store programs executed by control device 11 and various data used by control device 11. Storage device 12 may be a known recording medium such as magnetic recording medium or semiconductor recording medium, or a combination of multiple recording media. Alternatively, a portable recording medium that can be attached to or detached from playback control system 10, or a recording medium that can be written to or read from via a communication network (e.g., cloud storage) may be used as storage device 12.

[0032] Storage device 12 stores music data M for each piece of music, which specifies the time sequence of multiple notes that constitute the music. Figure 2This is a schematic diagram of the music data M. The music data M includes reference data R and performance data D. Reference data R specifies the time sequence of notes in the performance part played by the user U. Specifically, reference data R specifies the pitch and duration of each note in the performance part. On the other hand, performance data D specifies the time sequence of notes in the playback part played by the playback device 20. Specifically, performance data D specifies the pitch and duration of each note in the playback part. Reference data R and performance data D are, for example, time sequence data in MIDI (Musical Instrument Digital Interface) format, arranged in a time sequence, consisting of instruction data indicating the sounding or muting of musical notes and time data specifying the timing of the action indicated by the instruction data. Instruction data, for example, specifies pitch and intensity to indicate actions such as sounding or muting. Time data, for example, specifies the interval between consecutive instruction data. The period from indicating the sounding of a specific pitch through instruction data to indicating the muting of that pitch through subsequent instruction data is the sounding period associated with that pitch. Performance data D is an example of "sound data" representing a time sequence of multiple notes. The multiple notes in the playing part represented by performance data D are an example of "multiple notes" represented by "sound data".

[0033] Figure 1 The pickup device 13 picks up musical tones emitted from the instrument 200 by the user U's playing, generating an acoustic signal Z representing the waveform of that musical tone. For example, a microphone is used as the pickup device 13. For convenience, the A / D converter that converts the acoustic signal Z generated by the pickup device 13 from analog to digital is omitted from the illustration. Furthermore, in the first embodiment, a structure where the pickup device 13 is mounted on the playback control system 10 is shown, but the pickup device 13, which is separate from the playback control system 10, can also be connected to the playback control system 10 via wired or wireless means. Additionally, the output signal from an electric musical instrument such as an electric string instrument can also be received by the playback control system 10 as the acoustic signal Z. As understood from the above description, the pickup device 13 can be omitted from the playback control system 10.

[0034] Operating device 14 is an input device that receives instructions from user U. For example... Figure 3As illustrated, the operating device 14 of the first embodiment has a movable part 141 that can be moved by operation performed by the user U. The movable part 141 is an operating pedal that the user U can operate with his / her foot. For example, a pedal-type MIDI controller is used as the operating device 14. The user U can operate the operating device 14 at a desired moment in parallel with playing the instrument 200 with both hands. Alternatively, a touch panel that detects the user U's touch can also be used as the operating device 14.

[0035] The operating device 14 changes from one state (released) to another (operational state) corresponding to the operation performed by the user U. The released state is the state in which the operating device 14 is not operated by the user U. Specifically, the released state is the state in which the user U does not step on the movable part 141. The released state also means that the movable part 141 is in position H1. On the other hand, the operating state is the state in which the operating device 14 is operated by the user U. Specifically, the operating state is the state in which the user U steps on the movable part 141. The operating state also means that the movable part 141 is in position H2, which is different from position H1. The released state is an example of "first state," and the operating state is an example of "second state."

[0036] Figure 4 This is a block diagram illustrating the functional structure of the playback control system 10. The control device 11 executes a program stored in the storage device 12, thereby realizing multiple functions (performance analysis unit 31, playback control unit 32, and instruction receiving unit 33) for controlling the playback of the playback part executed by the playback device 20.

[0037] The performance analysis unit 31 analyzes the acoustic signal Z supplied from the pickup device 13 to estimate the performance position X within the music. The performance position X is the moment when the user U is playing within the music. The estimation of the performance position X is performed repeatedly in parallel with the playing of the part by the user U and the playback of the part by the playback device 20. That is, the performance position X is estimated at multiple moments on the timeline. The performance position X moves further back within the music as time passes.

[0038] Specifically, the performance analysis unit 31 calculates the performance position X by comparing the reference data R of the musical data M with the sound signal Z. Known analytical techniques (musical score alignment techniques) can be arbitrarily employed in estimating the performance position X using the performance analysis unit 31. For example, the analytical technique disclosed in Japanese Patent Application Publication No. 2016-099512 is used for estimating the performance position X. Furthermore, the performance analysis unit 31 can use statistical estimation models such as deep neural networks or hidden Markov models to estimate the performance position X.

[0039] The playback control unit 32 causes the playback device 20 to play each note specified by the performance data D. That is, the playback control unit 32 causes the playback device 20 to automatically play the played parts. Specifically, the playback control unit 32 moves backward over time from the position (hereinafter referred to as the "playing position") Y within the music, sequentially supplying the playback device 20 with the instruction data corresponding to the playing position Y from the performance data D. In other words, the playback control unit 32 functions as a sequencer that sequentially supplies the instruction data contained in the performance data D to the playback device 20. The process by which the playback control unit 32 causes the playback device 20 to play the played parts is performed in parallel with the performance of the played parts by the user U.

[0040] The playback control unit 32, in accordance with the result of the estimation of the performance position X by the performance analysis unit 31, causes the playback of the playback part through the playback device 20 to follow the performance of the music by the user U. That is, the automatic performance of the playback part through the playback device 20 proceeds at the same rhythm as the performance of the playback part by the user U. For example, if the progress of the performance position X (i.e., the performance speed of the user U) is fast, the playback control unit 32 increases the speed of the playback position Y (the playback speed of the playback device 20), and if the progress of the performance position X is slow, it decreases the speed of the playback position Y. In other words, the automatic performance of the playback part is executed at the same performance speed as the performance by the user U, so as to synchronize with the progress of the performance position X. Therefore, the user U can perform the playback part with the feeling that the playback device 20 is matching his own performance.

[0041] As described above, in the first embodiment, the playback of multiple notes in the playing part follows the performance of the instrument 200 by the user U, thus enabling the user U's intention (e.g., performance) or preferences to be appropriately reflected in the playback of the playing part.

[0042] The instruction receiving unit 33 receives a first instruction Q1 and a second instruction Q2 from the user U. The first instruction Q1 and the second instruction Q2 are generated by the user U's operation of the operating device 14. The first instruction Q1 is an instruction to temporarily stop the playback of the sound segment being played through the playback device 20. The second instruction Q2 is an instruction to resume the playback of the sound segment that was stopped by the first instruction Q1.

[0043] Specifically, the instruction receiving unit 33 receives the operation of the user U to switch the operating device 14 from the released state to the operating state as the first instruction Q1. That is, the user U provides the first instruction Q1 to the playback control system 10 by stepping on the movable part 141 of the operating device 14. For example, the instruction receiving unit 33 determines the moment when the movable part 141 begins to move from position H1 (released state) to position H2 (operating state) as the moment of the first instruction Q1. Furthermore, it is envisioned that the instruction receiving unit 33 determines the moment when the movable part 141 reaches a position intermediate from position H1 to position H2 as the moment of the first instruction Q1, or the instruction receiving unit 33 determines the moment when the movable part 141 reaches position H2 as the moment of the first instruction Q1.

[0044] Furthermore, the instruction receiving unit 33 receives the operation of the user U changing the operating device 14 from the operating state to the released state as the second instruction Q2. That is, the user U releases the movable part 141 of the operating device 14 from the state of stepping in, thereby giving the second instruction Q2 to the playback control system 10. For example, the instruction receiving unit 33 determines the moment when the movable part 141 begins to move from position H2 (operating state) to position H1 (released state) as the moment of the second instruction Q2. In addition, the following structure is also envisioned: the instruction receiving unit 33 determines the moment when the movable part 141 reaches the midpoint between position H2 and position H1 as the moment of the second instruction Q2, or the instruction receiving unit 33 determines the moment when the movable part 141 reaches position H1 as the moment of the second instruction Q2.

[0045] User U can assign the first instruction Q1 and the second instruction Q2 at any time during the performance of the playing part. Therefore, the interval between the first instruction Q1 and the second instruction Q2 is a variable length corresponding to the user U's intention. For example, user U assigns the first instruction Q1 before the start of a rest in the piece, and assigns the second instruction Q2 when user U has completed the rest of the desired length.

[0046] Figure 5 This is an explanatory diagram illustrating the relationship between the playback of the played part via the playback device 20 and the first instruction Q1 and the second instruction Q2. The duration of each note's articulation specified by the performance data D and the duration of each note's articulation actually played by the playback device 20 are shown in the diagram. Figure 5 It is recorded together in the middle.

[0047] Figure 5The note N1 is the note corresponding to the first instruction Q1 among a plurality of notes specified by the performance data D. Specifically, note N1 is the note being played by the playback device 20 at the moment of the first instruction Q1, among a plurality of notes in the playing part. After the first instruction Q1 is generated, the playback control unit 32 causes the playback device 20 to continue playing the note N1 until the end of the playing period specified by the performance data D for note N1. For example, at the end of the playing period of note N1, the playback control unit 32 supplies instruction data to the playback device 20 indicating the silencing of note N1. As understood from the above explanation, the playback of note N1 does not stop immediately at the moment of the first instruction Q1, but continues after the first instruction Q1 is generated until the end specified by the performance data D. Furthermore, note N1 is an example of "the first note".

[0048] Figure 5 Note N2 is the note immediately following note N1 among multiple notes specified by performance data D. After the playback of note N1 stops, the playback control unit 32, triggered by the second instruction Q2 generated by the user U, starts the playback device 20 to play note N2. That is, regardless of the starting position of the sounding period specified by performance data D for note N2, or the time interval between notes N1 and N2 specified by performance data D, the playback of note N2 is started conditionally based on the generation of the second instruction Q2. Specifically, when the instruction receiving unit 33 receives the second instruction Q2, the playback control unit 32 supplies the instruction data for note N2 in performance data D to the playback device 20. Therefore, the playback of note N2 begins immediately following the second instruction Q2. Furthermore, note N2 is an example of a "second note".

[0049] Figure 6 This is a flowchart illustrating the specific sequence of actions (hereinafter referred to as "playback control processing") Sa performed by the control device 11 on the playback device 20. The playback control processing Sa begins upon an instruction from the user U.

[0050] If playback control processing Sa is initiated, the control device 11 determines whether the waiting data W is in a valid state (Sa1). The waiting data W is data (e.g., a flag) indicating that playback of the playing part has been temporarily stopped by the first instruction Q1, and is stored in the storage device 12. Specifically, the waiting data W is set to a valid state (e.g., W=1) when the first instruction Q1 is generated, and set to an invalid state (e.g., W=0) when the second instruction Q2 is generated. In other words, the waiting data W is data indicating the state of waiting for playback of the playing part to resume.

[0051] If the waiting data W is not valid (Sa1: NO), the control device 11 (performance analysis unit 31) analyzes the audio signal Z supplied from the pickup device 13 to estimate the performance position X (Sa2). Accordingly, the control device 11 (playback control unit 32) causes the playback of the played part performed by the playback device 20 to proceed (Sa3) based on the estimated performance position X. That is, the control device 11 controls the playback of the played part performed by the playback device 20 to follow the performance of the played part performed by the user U.

[0052] The control device 11 (instruction receiving unit 33) determines whether the first instruction Q1 has been received from the user U (Sa4). If the first instruction Q1 has been received (Sa4: YES), the control device 11 (playback control unit 32) causes the playback device 20 to continue playing the note N1 that is playing at the time the first instruction Q1 is received until the end of the sounding period specified by the performance data D (Sa5). Specifically, the control device 11 moves the playback position Y at the same speed (rhythm) as the time when the first instruction Q1 is generated, and when the playback position Y reaches the end of the sounding period of the note N1, it supplies the playback device 20 with instruction data indicating that the note N1 has been muted. If the above processing is performed, the control device 11 changes the waiting data W from an invalid state to an valid state (W=1) (Sa6). In addition, the waiting data W can also be updated before the execution of step Sa5 (Sa6).

[0053] If the waiting data W is set to a valid state, the result of the determination in step Sa1 becomes affirmative. When the waiting data W is valid (Sa1: YES), the estimation of the playing position X (Sa2), the playback control of the playing part (Sa3), and the processing related to note N1 (Sa4-Sa6) are not executed. That is, based on the first instruction Q1 from user U, the playback control of the playing part linked to the playing position X is stopped. Furthermore, if the first instruction Q1 is not received (Sa4: NO), the processing related to note N1 (Sa5, Sa6) is not executed.

[0054] The control device 11 (instruction receiving unit 33) determines whether the second instruction Q2 has been received from the user U (Sa7). If the second instruction Q2 has been received (Sa7: YES), the control device 11 (playback control unit 32) causes the playback device 20 to play the note N2 immediately following the note N1 (Sa8). Specifically, the control device 11 updates the playback position Y to the starting point of the note N2. That is, the playback of the part that was stopped by the first instruction Q1 is resumed by the second instruction Q2. The control device 11 changes the waiting data W from a valid state to an invalid state (W=0) (Sa9). As mentioned above, if the waiting data W is set to an invalid state, the result of the determination in step Sa1 becomes negative. Therefore, the estimation of the playing position X (Sa2) and the playback control of the part are restored based on the second instruction Q2 (Sa3). In addition, the update of the waiting data W can also be performed before performing step Sa8 (Sa8).

[0055] The control device 11 determines whether to end the playback of the audio segment executed by the playback device 20 (Sa10). For example, if the playback has ended at the end of the audio segment, or if the user U has indicated the end, the control device 11 determines to end the playback of the audio segment. If the playback of the audio segment is not to end (Sa10: NO), the control device 11 jumps to step Sa1 and repeats the process illustrated above (Sa1-Sa9). On the other hand, if the control device 11 determines to end the playback of the audio segment (Sa10: YES), the playback control process Sa ends.

[0056] As described above, in the first embodiment, the note N1 corresponding to the first instruction Q1 is played, and after the playback of note N1 stops, the playback of note N2, which immediately follows note N1, begins, triggered by the second instruction Q2 generated by the user U. Therefore, the interval between the playback of note N1 and note N2 (e.g., the length of a rest in a piece of music) can be changed accordingly with each moment of the first instruction Q1 and the second instruction Q2.

[0057] Furthermore, in the first embodiment, the playback of note N1, which is being played at the moment the first instruction Q1 is generated, continues until the end of note N1 specified by the performance data D after the first instruction Q1 is generated. Therefore, compared with the structure that stops the playback of note N1 at the moment the first instruction Q1 is generated, the playback of note N1 can be appropriately continued according to the content of the performance data D.

[0058] In the first embodiment, the user U operates the operating device 14, thereby changing the interval between notes N1 and N2 to an appropriate duration corresponding to the user U's intention or preference. Specifically, in the first embodiment, a first instruction Q1 is generated by switching the operating device 14 from a released state to an operating state. After this operating state is maintained, a second instruction Q2 is generated by switching the operating device 14 from the operating state to a released state at a desired time after the generation of the first instruction Q1. That is, the first instruction Q1 and the second instruction Q2 are generated through a series of operations involving switching the operating state from a released state to an operating state and then back to a released state. Therefore, compared to a structure where the operating device 14 needs to be switched from a released state to an operating state for each of the first and second instructions Q1, the user U's operation of the operating device 14 is simplified.

[0059] B: Implementation Method 2

[0060] The second embodiment will be described. Furthermore, in the embodiments illustrated below, elements that function the same as in the first embodiment are appropriately omitted using the same reference numerals as those used in the description of the first embodiment.

[0061] In the first embodiment, when the first instruction Q1 is generated, the playback position Y is moved at the same speed as the moment of the first instruction Q1, and the playback of the note N1 is stopped when the playback position Y reaches the end of the note N1. In the second embodiment, the playback control unit 32 controls the movement speed of the playback position Y after the first instruction Q1 is generated (i.e., the playback speed of the playing part) and the operating speed V1 of the movable unit 141 to be variable accordingly. The operating speed V1 is the speed at which the movable unit 141 moves from position H1 corresponding to the released state to position H2 corresponding to the operating state. For example, the operating speed V1 is the average of multiple speeds calculated during the period when the movable unit 141 moves from position H1 to position H2.

[0062] Figure 7 This is an explanatory diagram relating to the state of the operating device 14 in the second embodiment. (See diagram below.) Figure 7 As illustrated, the instruction receiving unit 33 receives the first instruction Q1 at the moment when the movable part 141 begins to move from position H1 toward position H2. The playback control unit 32 controls the travel speed of the playback position Y after the first instruction Q1 is generated in accordance with the operating speed V1 of the movable part 141.

[0063] Specifically, the faster the operation speed V1 of the playback control unit 32, the faster the travel speed of the playback position Y increases. For example, as Figure 7As illustrated, when the operating speed V1 is V1_H, the speed of movement of the playback position Y exceeds the speed of movement of the playback position Y when the operating speed V1 is V1_L (V1_L < V1_H). Therefore, the faster the operating speed V1, the shorter the duration of the note N1. For example, the duration of the note N1 when the operating speed V1 is V1_H is shorter than the duration of the note N1 when the operating speed V1 is V1_L.

[0064] The second embodiment achieves the same effect as the first embodiment. In the second embodiment, the duration of note N1 is controlled accordingly with the operating speed V1, allowing the user U to adjust the duration of note N1. Furthermore, in the second embodiment, the operating device 14 for assigning the first indication Q1 and the second indication Q2 is also used for adjusting the duration of note N1. Therefore, compared to a structure where the user U operates a separate device for assigning the first indication Q1 and the second indication Q2 and adjusting the duration of note N1, the operation performed by the user U is simplified.

[0065] C: Third Implementation

[0066] In the first embodiment, the playback of note N2 begins immediately following the second instruction Q2. In the second embodiment, the time from the second instruction Q2 to the start of playback of note N2 (hereinafter referred to as the "delay time") is controlled to be variable accordingly with respect to the operating speed V2. The operating speed V2 is the speed at which the movable part 141 moves from position H2 corresponding to the operating state toward position H1 corresponding to the released state. For example, the operating speed V2 is the average of multiple speeds calculated during the period when the movable part 141 moves from position H2 to position H1.

[0067] Figure 8 This is an explanatory diagram relating to the state of the operating device 14 in the third embodiment. (See diagram below.) Figure 8 As illustrated, the instruction receiving unit 33 receives the second instruction Q2 at the moment when the movable part 141 begins to move from position H2 toward position H1. The playback control unit 32 controls the delay time and operating speed V2 to be variable accordingly.

[0068] Specifically, the faster the operation speed V2 of the playback control unit 32, the shorter the delay time. For example, as Figure 8 As illustrated, the delay time when the operating speed V2 is V2_L is longer than the delay time when the operating speed V2 is V2_H (V2_H > V2_L). Therefore, the slower the operating speed V2, the later the start time of the playback of note N2 will be on the timeline.

[0069] The same effect as in the first embodiment is achieved in the third embodiment. In the third embodiment, the starting time of the playback of note N2 is controlled accordingly with the operating speed V2, thus allowing the user U to adjust the starting point of the initial note N2 after the playback part is resumed. Furthermore, in the third embodiment, the operating device 14 for assigning the first instruction Q1 and the second instruction Q2 also serves as the adjustment of the starting point of note N2. Therefore, compared to a structure where the user U operates a separate device for assigning the first instruction Q1 and the second instruction Q2 and adjusting the starting point of note N2, there is the advantage of simplified operation by the user U. Moreover, the structure of the second embodiment can also be applied to the third embodiment.

[0070] D: Implementation Method 4

[0071] Figure 9 This is a block diagram illustrating the functional structure of the playback control system 10 in the fourth embodiment. The control device 11 in the fourth embodiment functions as an editing processing unit 34 in addition to the same elements as in the first embodiment (performance analysis unit 31, playback control unit 32, and instruction receiving unit 33). The editing processing unit 34 edits the performance data D stored in the storage device 12 in accordance with the instructions from the user U. The operation of the elements other than the editing processing unit 34 is the same as in the first embodiment. Therefore, the same effect as in the first embodiment is achieved in the fourth embodiment. Furthermore, the structure of the second or third embodiment can also be applied to the fourth embodiment.

[0072] Figure 10 This is an explanatory diagram of the operation of the editing and processing unit 34. Figure 10 The diagram shows notes N1 and N2 specified by the performance data D of the playing voice. Similar to the methods described above, the first instruction Q1 and the second instruction Q2 are generated by the user U at any given time. Therefore, a time difference L is generated between the starting point of note N2 specified by the performance data D and the moment of the second instruction Q2. The editing processing unit 34 edits the performance data D in a manner that reduces the time difference L.

[0073] Figure 11 This is a flowchart illustrating the specific sequence of the editing process (hereinafter referred to as "editing process") Sb performed by the editing processing unit 34 on the performance data D. For example, the editing process Sb is performed when the playback of the playback part via the playback device 20 (the aforementioned playback control process Sa) has been repeated a predetermined number of times. Alternatively, the editing process Sb can also be initiated based on an instruction from the user U.

[0074] If editing process Sb begins, the editing processing unit 34 calculates the dispersion Δ of the time differences L in the past playback control processes Sa (Sb1). The dispersion Δ is a statistic that represents the degree of dispersion associated with multiple time differences L. For example, the dispersion, standard deviation, or distribution range of multiple time differences L are used as the dispersion Δ.

[0075] The editing unit 34 determines whether the dispersion Δ exceeds the threshold Δth (Sb2). If the dispersion Δ exceeds the threshold Δth, it is presumed that the user U is consciously changing the waiting time until the playback of note N2 resumes while practicing playing the piece. Therefore, it is inappropriate to edit the performance data D corresponding to the multiple time differences L. On the other hand, if the dispersion Δ is below the threshold Δth, it is presumed that the multiple time differences L are values ​​according to the user U's intention or preference (i.e., the user U's inherent preferred values).

[0076] Considering the above tendencies, when the dispersion Δ is lower than the threshold Δth (Sb2: NO), the editing processing unit 34 edits the performance data D accordingly (Sb3-Sb4) based on the multiple time differences L. On the other hand, when the dispersion Δ exceeds the threshold Δth (Sb2: YES), the editing processing unit 34 does not perform the editing of the performance data D (Sb3, Sb4) but ends the editing process Sb.

[0077] In editing the performance data D, the editing processing unit 34 calculates an average time difference La (Sb3) by averaging multiple time differences L. Furthermore, the editing processing unit 34 changes the starting point of the note N2 specified by the performance data D by the average time difference La (Sb4). For example, if the average time difference La is negative, the editing processing unit 34 moves the starting point of the note N2 specified by the performance data D forward by a time equivalent to the average time difference La. Conversely, if the average time difference La is positive, the editing processing unit 34 moves the starting point of the note N2 specified by the performance data D backward by a time equivalent to the average time difference La. In other words, if the user U tends to ensure sufficient waiting time before the note N2, the starting point of the note N2 specified by the performance data D is changed backward; if the user tends to have a shorter waiting time, the starting point of the note N2 specified by the performance data D is changed forward.

[0078] As understood from the above description, in the fourth embodiment, the performance data D is edited in accordance with the time difference L in the performance of the playing part by the user U. Therefore, it is possible to reflect the inherent tendencies of each user U in the performance data D.

[0079] E: Fifth Implementation

[0080] Figure 12 This is a block diagram illustrating the structure of the playback system 100 according to the fifth embodiment. The playback system 100 includes a playback control system 10 and a performance device 50. The playback control system 10 of the fifth embodiment includes a display device 15 based on the same elements (control device 11, storage device 12, pickup device 13, and operation device 14) as the playback control system 10 of the first embodiment. The display device 15 displays an image indicated by the control device 11. The display device 15 is, for example, a liquid crystal display panel or an organic EL display panel.

[0081] The playing device 50 functions not only as a playback device that automatically plays the playing part of a musical piece, but also as an instrument that can be manually played by the user U1. Specifically, the playing device 50, like the playback device 20 in the aforementioned embodiments, has a drive mechanism 21 and a sound-producing mechanism 22. Furthermore, the playing device 50 of the fifth embodiment can be used as the playback device 20 of the first to fourth embodiments.

[0082] The user U1 performs manual playing, such as playing by pressing keys on a keyboard or through body movements. The sound-producing mechanism 22 operates in conjunction with the playing by the user U1, thereby emitting musical sounds from the playing device 50. Furthermore, the playing device 50 and the playing by the user U1 simultaneously output instruction data d, indicating the playing, to the playback control system 10. The instruction data d, for example, specifies pitch and intensity, thus specifying actions such as producing sound or muting. On the other hand, the user U2 plays an instrument 200. The instrument 200 is a natural instrument such as a stringed instrument that produces sound through playing by the user U.

[0083] Figure 13 This is a block diagram illustrating the functional structure of the playback control system 10 in the fifth embodiment. The control device 11 in the fifth embodiment, by executing a program stored in the storage device 12, functions as a preparation processing unit 35 in addition to the same elements as in the first embodiment (performance analysis unit 31, playback control unit 32, and instruction receiving unit 33). The preparation processing unit 35 generates music data M (performance data D and reference data R) used in the playback control processing Sa. Specifically, the preparation processing unit 35 generates music data M corresponding to the performance of the performance device 50 by user U1 and the performance of the instrument 200 by user U2. The preparation processing unit 35 includes a first recording unit 41, a second recording unit 42, and a reference data generation unit 43.

[0084] The reference data generation unit 43 generates reference data R used in the playback control processing Sa. Specifically, the reference data generation unit 43 performs a reference data generation process Sc before starting the playback control processing Sa. Figure 14 The reference data R generated by the reference data generation unit 43 is stored in the storage device 12.

[0085] During the period before the reference data generation process Sc is executed (hereinafter referred to as the "preparation period"), users U1 and U2 play a musical piece together. Specifically, user U1 plays the playback part of the musical piece through the playing device 50, and user U2 plays the playback part of the musical piece through the instrument 200. The reference data generation process Sc is a process that generates reference data R based on the results of the musical piece played by users U1 and U2 during the preparation period.

[0086] The first recording unit 41 acquires performance data D, which specifies multiple notes indicated by the user U1 to the performance device 50 during the performance in preparation. Specifically, the first recording unit 41 generates performance data D, which is arranged in time sequence with instruction data d supplied sequentially from the performance device 50 in accordance with the performance performed by the user U1, and time data specifying the interval between consecutive instruction data d. The first recording unit 41 stores the performance data D in the storage device 12. The performance data D stored in the storage device 12 is used for playback control processing Sa, as illustrated in the aforementioned embodiments. Furthermore, the performance data D acquired by the first recording unit 41 can also be edited by the editing processing unit 34 illustrated in the fourth embodiment.

[0087] The second recording unit 42 acquires the acoustic signal Z (hereinafter referred to as "reference signal Zr1") generated by the pickup device 13 during the preparation period. During the preparation period, in addition to the musical tones emitted from the instrument 200 by the performance by the user U2, musical tones emitted from the playing device 50 by the performance by the user U1 also reach the pickup device 13. Therefore, the reference signal Zr1 is a signal representing the mixed tone of the musical tones of the instrument 200 and the playing device 50. As understood from the above description, the second recording unit 42 acquires the reference signal Zr1, which represents the musical tones emitted by the instrument 200 by the performance of the piece of music (first performance) using the instrument 200 and the musical tones emitted by the playing device 50 by the performance of the instrument using the playing device 50. The second recording unit 42 stores the reference signal Zr1 in the storage device 12. As understood from the above description, the performance data D and the reference signal Zr1 are stored in the storage device 12 during the preparation period.

[0088] The reference data generation unit 43 generates reference data R by utilizing the reference data generation process Sc of the reference signal Zr1 obtained by the second recording unit 42. Furthermore, in the aforementioned embodiments, MIDI data arranged in a time sequence of instruction data and time data is exemplified as reference data R. The reference data R in the fifth embodiment specifies the performance period, sounding moment, and pitch transition within a musical piece. The performance period is the period during which the user U2 plays the instrument 200 during the preparation period. In other words, the performance period is the period during which the instrument 200 emits musical notes. For example, the reference data R specifies the start and end times of the performance period. The sounding moment is the moment (i.e., the starting point) at which each musical note in the musical piece begins to be sounded during the preparation period through the performance by the user U2. For example, the sounding moment is specified using the reference data R. Pitch transition is the time sequence of the pitch (interval) of the musical notes emitted by the instrument 200 through the performance by the user U1 during the preparation period.

[0089] Figure 14 This is a flowchart illustrating the specific sequence of the reference data generation process Sc. After obtaining the performance data D and reference signal Zr1 during the preparation period, the reference data generation process Sc begins, for example, triggered by an instruction from the user U (U1, U2) to the operating device 14.

[0090] If reference data generation processing Sc begins, the reference data generation unit 43 generates a reference signal Zr2 (Sc1) by emphasizing the acoustic components of the musical tone emitted from the instrument 200 in the reference signal Zr1. As mentioned earlier, the reference signal Zr1 includes acoustic components of the musical tone emitted from the playing device 50 and the musical tone emitted from the instrument 200. The reference data generation unit 43 generates the reference signal Zr2 by suppressing the acoustic components of the musical tone from the playing device 50 in the reference signal Zr1.

[0091] For example, the reference data generation unit 43 generates the reference signal zr2 by subtracting the amplitude spectrum of the musical tone from the amplitude spectrum of the reference signal Zr1. The amplitude spectrum of the musical tone from the playing device 50 is generated, for example, by known sound source processing and frequency analysis such as discrete Fourier transform of the musical tone signal representing the musical tone specified by the playing data D. The degree to which the amplitude spectrum of the musical tone from the playing device 50 is subtracted is adjusted accordingly to the instructions from the user U to the operating device 14.

[0092] Furthermore, the process of generating reference signal Zr2 based on reference signal Zr1 is not limited to the examples above. For example, known sound source separation techniques can be used to emphasize the acoustic component of the musical instrument 200 in reference signal Zr1. Additionally, if, for example, the musical sound emitted from the playing device 50 is difficult to reach the pickup device 13, step Sc1 can be omitted. In the case where step Sc1 is omitted, the following processes are performed on reference signal Zr1 instead of reference signal Zr2. Reference signal Zr1 and reference signal Zr2 are examples of a "first acoustic signal".

[0093] Control device 11 and reference data generation processing Sc work in parallel. Figure 15 The preparation screen 60 is displayed on the display device 15. The waveform 61 of the reference signal Zr2 is displayed on the preparation screen 60. The user U can adjust the range of the reference signal Zr2 displayed on the preparation screen 60 by operating the operation device 14.

[0094] like Figure 14 As illustrated, the reference data generation unit 43 determines one or more performance periods in the reference signal Zr2 by analyzing the reference signal Zr2 (Sc2). When determining the performance period, a first HMM (Hidden Markov Model) is used to estimate the performance period in accordance with the intensity of the reference signal Zr2. In general, the period in the reference signal Zr2 where the signal intensity exceeds a threshold τ is determined as the performance period. However, the method for determining the performance period is not limited to the example described above.

[0095] The performance period determined from the reference signal Zr2 is displayed in the preparation screen 60. Specifically, the control device 11, as shown in the image, is... Figure 15 As illustrated, the portion 61a during the performance period and the portion 61b outside the performance period of the waveform 61 of the reference signal Zr2 are displayed in different ways. "Display mode" refers to the characteristics of an image that an observer can visually recognize. For example, in addition to the three attributes of color—hue (tone), chroma, and lightness (grayscale)—patterns or shapes are also included in the concept of "display mode." Furthermore, the method of displaying the performance period is not limited to the example above.

[0096] Additionally, the preparation screen 60 displays an operation image 62 that the user U can operate via the operation device 14. The reference data generation unit 43 controls the threshold τ applied to determine the performance period in accordance with the instructions from the user U regarding the operation image 62. The smaller the threshold τ set by the user U, the easier it is to determine each moment of the reference signal Zr2 as a moment within the performance period.

[0097] The reference data generation unit 43 determines multiple articulation moments in the reference signal Zr2 by analyzing the reference signal Zr2 (Sc3). For example... Figure 15 As illustrated, in the preparation screen 60, an indicator image 63 representing the sounding time determined from the reference signal Zr2 is displayed. The indicator image 63 is a vertical line arranged on the time axis for each sounding time. However, the specific arrangement of the indicator image 63 is not limited to the example above.

[0098] When determining the performance time via the reference data generation unit 43, a second HMM consisting of multiple states corresponding to different pitches is used. The moment when the transition occurs between different states (i.e., the moment of pitch change) is determined as the performance time. Furthermore, corresponding to the indication of the type of instrument 200 issued by the user U, the reference data generation unit 43 can limit the range of state transitions (i.e., the range of pitch variation) in the second HMM to the range of the instrument 200. That is, transitions to states corresponding to pitches outside the range of the instrument 200 are prohibited.

[0099] For each state of the second HMM, a first transition matrix Λ1 and a second transition matrix Λ2 are defined. The first transition matrix Λ1 and the second transition matrix Λ2, associated with each state, are matrices that define the transition probability (self-transition probability) to that state and the transition probability to other states. The values ​​of the transition probabilities differ between the first transition matrix Λ1 and the second transition matrix Λ2. Specifically, the transition probability to other states in the first transition matrix Λ1 is lower than the transition probability to other states in the second transition matrix Λ2. In other words, the self-transition probability in the first transition matrix Λ1 is higher than the self-transition probability in the second transition matrix Λ2. Therefore, when the first transition matrix Λ1 is applied to the second HMM, compared to when the second transition matrix Λ2 is applied, state transitions are less likely to occur. That is, the frequency of moments on the time axis that are presumed to be phonation moments decreases.

[0100] The reference data generation unit 43 of the fifth embodiment applies a transformation matrix Λ calculated for each state by weighting the first transformation matrix Λ1 and the second transformation matrix Λ2 to each state of the second HMM. The transformation matrix Λ is represented, for example, by the following formula (1). The coefficient α of formula (1) is a value in the range of 0 or more and 1 or less.

[0101] Λ=α·Λ1+(1-α)·Λ2 (1)

[0102] like Figure 15As illustrated, in the preparation screen 60, an operation image 64 operable by the user U via the operation device 14 is displayed. The reference data generation unit 43 controls the coefficient α applied to determine the playing time in accordance with the instructions from the user U regarding the operation image 64. As understood according to formula (1), the larger the coefficient α, the greater the influence of the first conversion matrix Λ1 on the conversion matrix Λ, and the less the influence of the second conversion matrix Λ2 on the conversion matrix Λ. Therefore, the transition between states in the second HMM becomes difficult to occur. That is, the probability that each moment of the reference signal Zr2 is judged as a playing moment decreases. As understood according to the above explanation, there is a tendency that the larger the coefficient α, the fewer the number of playing moments determined with respect to the reference signal Zr2. In other words, the smaller the coefficient α, the more the number of playing moments increases. The user U checks the preparation screen 60 while operating the operation image 64, thereby adjusting the coefficient α so that the number of playing moments displayed by the instruction image 63 is appropriate. Furthermore, the methods for determining the timing of pronunciation are not limited to the examples above.

[0103] The reference data generation unit 43 determines the pitch transition by analyzing the reference signal Zr2 (Sc4). When determining the pitch transition, a estimation model is used, for example, to learn the relationship between the frequency characteristics of the reference signal Zr2 and the pitch transition through machine learning. The estimation model is, for example, a deep neural network such as a convolutional neural network or a recurrent neural network. Control data, including the spectrum generated by a constant Q-transform of the reference signal Zr2, is supplied to the estimation model. The estimation model outputs a time sequence of pitch (i.e., the pitch transition) related to the control data. Furthermore, the method for determining the pitch transition is not limited to the examples above. Additionally, the order of emphasizing the acoustic components of the musical tone (Sc1), determining the performance period (Sc2), determining the sounding time (Sc3), and determining the pitch transition (Sc4) can be arbitrarily changed. For example, the acoustic components of the musical tone from the playing device 50 can be suppressed from the reference signal Zr1 using a value calculated in the determination of the pitch transition (Sc4) (e.g., the post-hoc probability of the pitch).

[0104] The reference data generation unit 43 determines whether an instruction has been received from the user U (Sc5). Specifically, it determines whether the user U has indicated a change in the threshold τ or the coefficient α. If an instruction has been received from the user U (Sc5: YES), the reference data generation unit 43 determines the performance period, the timing of articulation, and the pitch transition by applying the changed threshold τ or coefficient α (Sc1 to Sc4).

[0105] Furthermore, it is envisioned that the processing load for determining the timing of articulation exceeds the processing load for determining the performance period. Considering the above, a structure is envisioned that performs the determination of the timing of articulation at a lower frequency compared to determining the performance period. For example, the reference data generation unit 43 repeatedly performs the determination of the performance period (Sc2) during the process where the user U changes the threshold τ through the operation image 62. For example, the determination of the performance period is repeatedly performed during the period when the user U drags the operation image 62 to change the threshold τ (Sc2). On the other hand, the reference data generation unit 43 does not perform the determination of the timing of articulation during the process where the user U changes the coefficient α through the operation image 64, but performs the determination of the timing of articulation at the end of the change of coefficient α (i.e., the determination of the changed coefficient α) (Sc3). For example, the timing of articulation is determined at the stage when the user U's dragging of the operation image 62 to change the coefficient α ends. According to the above structure, there are advantages that the performance period changes rapidly according to the user U's instructions, and on the other hand, the processing load required for determining the timing of articulation is reduced.

[0106] If no instruction is received from user U (Sc5: NO), the reference data generation unit 43 determines whether a save instruction for reference data R has been received from user U (Sc6). If no instruction is received (Sc6: NO), the reference data generation unit 43 proceeds to step Sc5. On the other hand, if a save instruction is received from user U (Sc6: YES), the reference data generation unit 43 saves the reference data R, which specifies the performance period, sounding time, and pitch transition at the current moment, to the storage device 12 (Sc7).

[0107] The reference data R generated by the reference data generation process Sc illustrated above is applied to the playback control process Sa. The specific order of the playback control process Sa in the fifth embodiment is the same as in the aforementioned embodiments. For example, the performance analysis unit 31 compares the sound signal Z representing the musical tone emitted by the instrument 200 by the performance of the playing part (second performance) performed by the user U2 with the reference data R generated by the reference data generation process Sc, thereby estimating the performance position X in parallel with the performance. In addition, the playback control unit 32 causes the performance device 50 to play each note of the playing part specified by the performance data D obtained by the first recording unit 41. Specifically, the playback control unit 32, in accordance with the estimation result obtained by the performance analysis unit 31, causes the playback of each note of the playing part to follow the performance performed by the user U2.

[0108] Furthermore, in the above description, a single playing device 50 was used for acquiring the performance data D during preparation and for automatic performance via playback control processing Sa. However, a playback device 20 that performs automatic performance via playback control processing Sa can be used independently of the playing device 50 used for acquiring the performance data D during preparation. That is, the automatic performance function is not necessary in the playing device 50. Alternatively, different instruments 20 can be used for acquiring the reference signal Zr1 during preparation and for performance in parallel with playback control processing Sa.

[0109] Furthermore, similar to the aforementioned methods, the playback control unit 32 continues the playback of note N1, which is being played at the moment the user U2 generates the first instruction Q1, until the end of note N1 as specified by the performance data D, after the first instruction Q1 is generated. Additionally, after the playback of note N1 stops, the playback control unit 32 begins playback of note N2, which immediately follows note N1, triggered by the second instruction Q2 generated by the user U. Therefore, the fifth embodiment achieves the same effect as the aforementioned methods.

[0110] Furthermore, according to the fifth embodiment, reference data R is generated in accordance with the result of the user U2 playing the instrument 200. Therefore, it is possible to generate reference data R reflecting the user U2's performance for any piece of music for which no reference data R has been prepared.

[0111] The control device 11 in the fifth embodiment operates in parallel with the playback control processing Sa. Figure 16 The playback screen 70 is displayed on the display device 15. The playback screen 70 includes an operation screen 71, a playback screen 72, an instruction screen 73, an instruction screen 74, and an operation screen 75.

[0112] Operation image 71 is an image used by the user U to indicate the start of automatic performance (start of playback control processing Sa) via the performance device 50. Playback image 72 is an image indicating the current playback position Y involved in the performance device 50. Specifically, playback image 72 consists of a time axis 721 representing the entire interval of the music and an indication image 722 representing the current playback position Y. As the automatic performance progresses, the indication image 722 moves along the time axis 721.

[0113] The indicator image 73 is used to notify the user U of the pronunciation times specified by the reference data R. Specifically, the control device 11 changes the display mode of the indicator image 73 when the playback position Y reaches each pronunciation time specified by the reference data R. For example, the indicator image 73 is momentarily magnified when the playback position Y coincides with a pronunciation time. By visually recognizing the indicator image 73, the user U can confirm each pronunciation time specified by the reference data R. Therefore, relative to the user U2's performance during preparation, it is possible to visually confirm whether the pronunciation times estimated by the reference data generation process Sc are exceeded or exceeded (i.e., whether the processing for determining the pronunciation times is appropriate).

[0114] The indicator image 74 is used to inform the user U whether the volume of the musical tone emitted by the instrument 200 is appropriate for accurately estimating the playing position X. If the volume σ of the sound signal Z is too high or too low, there is a tendency for the estimation accuracy of the playing position X by the playing analysis unit 31 to decrease. Taking this tendency into consideration, the control device 11 changes the display mode of the indicator image 74 accordingly to the volume σ of the sound signal Z.

[0115] Specifically, when the volume σ of the audio signal Z is above the threshold σL and below the threshold σH, the control device 11 maintains the display mode of the indicator image 74 in the first mode. The threshold σH is a value that exceeds the threshold σL. The thresholds σL and σH are each set experimentally or statistically so that the playing position X can be estimated with target accuracy when the volume σ of the audio signal Z is within the range of above the threshold σL and below the threshold σH.

[0116] Furthermore, when the volume σ of the audio signal Z is lower than the threshold σL, the control device 11 changes the display mode of the indicator image 74 to a second mode, different from the first mode. That is, when the volume σ is too low for accurate estimation of the playing position X, the indicator image 74 is displayed in the second mode. On the other hand, when the volume σ of the audio signal Z exceeds the threshold σH, the control device 11 changes the display mode of the indicator image 74 to a third mode, different from the first mode. That is, when the volume σ is too high for accurate estimation of the playing position X, the indicator image 74 is displayed in the third mode. As understood from the above explanation, the decrease in the accuracy of the estimation of the playing position X is notified to the user U through the change in the display mode of the indicator image 74. Moreover, the differences between the second and third modes are irrelevant.

[0117] In the above description, the focus is on the volume σ of the audio signal Z, but the index that serves as the reference for changing the display mode of the indicator image 74 is not limited to the volume σ. For example, the control device 11 may control the display mode of the indicator image 74 in accordance with the likelihood of the pitch estimated in step Sc4 of the reference data generation process Sc.

[0118] Figure 16 Operation image 75 is used by the user U to adjust the playing speed of automatic playback. Regarding the playing speed, the user U can operate operation image 75 to indicate the playing speed of automatic playback via a multiple (i.e., a relative value) relative to a predetermined reference value. Playback control unit 32 uses the value indicated by the user U as the reference value for the playing speed to control the automatic playback performed by the playing device 50, so that the playback position Y follows the playing position X.

[0119] F: Variation

[0120] Below are examples of specific variations added to the methods illustrated above. Two or more methods selected arbitrarily from the examples below can be appropriately combined without contradiction.

[0121] (1) In the aforementioned methods, the operation of changing the operating device 14 from the released state to the operating state is received by the instruction receiving unit 33 as the first instruction Q1, but the method of the first instruction Q1 is not limited to the examples above. For example, a specific action performed by the user U can be detected and used as the first instruction Q1. When detecting the user U's action, various detection devices such as a camera or an accelerometer can be used. For example, the instruction receiving unit 33 can determine various actions of the user U, such as lifting one hand, lifting the instrument 200, or breathing (e.g., inhaling), and use them as the first instruction Q1. The user U's breathing is, for example, the breath (breathing) when playing a wind instrument as the instrument 200. The operating speed V1 in the second embodiment is generally expressed as the speed of the user U's action determined as the first instruction Q1.

[0122] Specific data representing the first instruction Q1 (hereinafter referred to as "first data") can be included in the performance data D. The first data is, for example, data representing fermal within the music. The instruction receiving unit 33 determines that the first instruction Q1 has been generated when the playback position Y reaches the first data. As understood from the above explanation, the first instruction Q1 is not limited to the instruction from the user U. Furthermore, if the dispersion Δ in the editing process Sb exceeds the threshold Δth, the editing processing unit 34 may also append the first data to the note N1.

[0123] (2) In the aforementioned embodiments, the operation of changing the operating device 14 from the operating state to the released state is received by the instruction receiving unit 33 as the second instruction Q2. However, the method of the second instruction Q2 is not limited to the examples above. For example, similar to the first instruction Q1 in the first embodiment, the operation of changing the operating device 14 from the released state to the operating state can also be received by the instruction receiving unit 33 as the second instruction Q2. That is, two operations, including stepping in and releasing the movable part 141, can be detected and used as the first instruction Q1 and the second instruction Q2.

[0124] Alternatively, specific actions of the user U can be detected and used as the second instruction Q2. When detecting the user U's actions, various detection devices can be used, such as a camera or an accelerometer. For example, the instruction receiving unit 33 can determine various actions of the user U, such as putting down one hand, lowering the instrument 200, or breathing (e.g., exhaling), and use these as the second instruction Q2. The user U's breathing is, for example, the breath (breathing) when playing a wind instrument as instrument 200. In the second embodiment, the operating speed V2 is generally expressed as the speed of the user U's actions determined as the second instruction Q2.

[0125] Specific data representing the second instruction Q2 (hereinafter referred to as "second data") can be included in the performance data D. The second data is, for example, data representing fermal in the music. The instruction receiving unit 33 determines that the second instruction Q2 has been generated when the playback position Y reaches the second data. As understood from the above description, the second instruction Q2 is not limited to instructions from the user U.

[0126] As illustrated above, imagine a structure where one action from a pair of users U is taken as the first instruction Q1, and the other as the second instruction Q2. For example, taking the action of user U raising one hand as the first instruction Q1, and taking the action of lowering one hand immediately following that action as the second instruction Q2. Similarly, taking the action of user U lifting the instrument 200 as the first instruction Q1, and taking the action of lowering the instrument 200 immediately following that action as the second instruction Q2. Likewise, taking the action of user U inhaling as the first instruction Q1, and taking the action of exhaling immediately following that action as the second instruction Q2.

[0127] However, the first instruction Q1 and the second instruction Q2 do not need to be the same type of action performed by the user U. That is, individual actions that the user U can perform independently can also be detected and used as the first instruction Q1 and the second instruction Q2. For example, the instruction receiving unit 33 can detect the operation of the operating device 14 as the first instruction Q1, and detect other actions such as lifting the instrument 200 or breathing as the second instruction Q2.

[0128] (3) In the foregoing embodiments, the automatically playing musical instrument is exemplified as the playback device 20, but the structure of the playback device 20 is not limited to the above examples. For example, a sound source system having a sound source device and a sound playback device may also be used as the playback device 20, wherein the sound source device generates an acoustic signal of musical tone in response to an instruction from the playback control system 10, and the sound playback device plays the musical tone represented by the acoustic signal. The sound source device is implemented as a hardware sound source or a software sound source. Furthermore, the same applies to the playing device 50 in the fifth embodiment.

[0129] (4) In the aforementioned methods, the playback device 20 is controlled using performance data D, which specifies the time sequence of multiple notes in a musical piece. However, the form of data used in controlling the playback device 20 is not limited to the examples above. For instance, waveform data representing the waveforms of multiple notes constituting the played part can also be used to control the playback device 20. The waveform data consists of a time sequence of multiple samples. The playback control unit 32 supplies the sample from the waveform data corresponding to the playback position Y to the playback device 20. The playback device 20 is a sound playback system that plays the sounds represented by the time sequence of samples supplied from the playback control system 10.

[0130] The waveform data includes additional articulation point data representing the starting points (hereinafter referred to as "articulation points") of multiple notes. If the instruction receiving unit 33 receives the first instruction Q1, the playback control unit 32 instructs the playback device 20 to play the note represented by the waveform data up to the moment immediately preceding the articulation point, which is located immediately following the first instruction Q1, and then waits for the second instruction Q2. That is, the playback of the note (first note) that is being played at the moment the first instruction Q1 is generated continues until the end of the first note. On the other hand, if the instruction receiving unit 33 receives the second instruction Q2, the playback control unit 32 instructs the playback device 20 to play the note (second note) represented by the waveform data starting from the articulation point immediately following the completion of the playback interval in the waveform data (located immediately following the first instruction Q1). That is, after the playback of the first note stops, the playback of the second note immediately following the first note begins, triggered by the second instruction Q2. Furthermore, in the above description, variations of the first to fourth embodiments having a playback device 20 are shown, but the form of the data used to control the automatic performance performed by the performance device 50 of the fifth embodiment is also not limited to performance data D; for example, waveform data shown above can also be used.

[0131] As understood from the above explanation, the performance data D exemplified in the aforementioned methods and the waveform data exemplified in the variant examples are collectively represented as tone data representing a time sequence of multiple notes.

[0132] (5) For example, the playback control system 10 can be implemented by a server device that communicates with terminal devices such as smartphones or tablets. The terminal device includes: a pickup device 13 that generates an audio signal Z corresponding to the performance by the user U; and a playback device 20 (or the performance device 50 in the fifth embodiment) that plays the music in accordance with the instructions from the playback control system 10. The terminal device sends the audio signal Z generated by the pickup device 13 and the first instruction Q1 and the second instruction Q2 corresponding to the user U's actions to the playback control system 10 via a communication network. The playback control system 10 causes the playback device 20 of the terminal device to play the playback part of the music in accordance with the performance position X estimated based on the audio signal Z and the first instruction Q1 and the second instruction Q2 received from the terminal device. In addition, a performance analysis unit 31 may also be mounted on the terminal device. The terminal device sends the performance position X estimated by the performance analysis unit 31 to the playback control system 10. In the above structure, the performance analysis unit 31 is omitted from the playback control system 10.

[0133] (6) The functions of the playback control system 10 illustrated above, as described above, are achieved through the coordinated operation of one or more processors constituting the control device 11 and the program stored in the storage device 12. The program involved in this invention can be provided and installed on a computer in the form of a computer-readable recording medium. The recording medium is, for example, a non-transitory recording medium, with optical recording media (optical discs) such as CD-ROMs being preferred, but it may also include any known form of recording medium such as semiconductor recording media or magnetic recording media. Furthermore, the term "non-transitory recording medium" includes any recording medium other than transient propagating signals, and does not exclude volatile recording media. In addition, in a configuration where the transmission device transmits the program via a communication network, the storage device storing the program in the transmission device is equivalent to the aforementioned non-transitory recording medium.

[0134] G: Postscript

[0135] Based on the examples above, for instance, understand the following structure.

[0136] One aspect (Aspect 1) of the present invention relates to a playback control method that plays multiple sounds using sound data representing a time sequence of multiple sounds. The method involves playing a first sound corresponding to a first instruction among the multiple sounds, and after the playback of the first sound stops, starting playback of a second sound immediately following the first sound, triggered by a second instruction generated by the user. According to this method, after the playback of the first sound corresponding to the first instruction stops, starting playback of the second sound immediately following the first sound, triggered by a second instruction from the user, allows for appropriate control of the interval (e.g., the duration of a rest) between the first and second sounds in accordance with each moment of the first and second instructions.

[0137] "Sound data" refers to any form of data that represents a time sequence of multiple notes. For example, performance data that specifies the duration of each note's articulation (specifically, articulation and muting) or waveform data representing the waveforms of multiple notes on a timeline are included in the concept of "sound data".

[0138] "First instruction" can be, for example, an instruction corresponding to the user's action or an instruction attached to the sound data. An instruction corresponding to the user's action is, for example, an instruction given by the user through operation of the operating device. Alternatively, a first instruction may be generated when the user performs a specific action (such as lifting an instrument or taking a breath while playing a wind instrument). Instructions attached to the sound data include, for example, various performance instructions such as Feuerbach's to indicate the duration of a note or rest.

[0139] "Second instruction" is, for example, an instruction corresponding to the user's action. An instruction corresponding to the user's action is, for example, an instruction given by the user through an operation performed on the operating device. Alternatively, the second instruction may be generated when the user performs a specific action (such as lifting a musical instrument or taking a breath while playing a wind instrument).

[0140] "Starting the playback of the second note based on the second instruction" means starting the playback of the second note with the second instruction as a condition, regardless of the relationship between the time of the second instruction and the time of the start of the playback of the second note. For example, in addition to starting the playback of the second note at or immediately after the second instruction, starting the playback of the second note at a time after a predetermined period of time has elapsed from the second instruction is also included in the method of "starting the playback of the second note based on the second instruction".

[0141] The "playing" of multiple notes means radiating each note as a sound wave. Therefore, for example, the automatic playing of multiple notes by an automatic playing instrument such as an automatic playing piano, or the reproduction of multiple notes by a sound source device and a sound reproduction device, are included in the concept of "playing".

[0142] In a specific example of Method 1 (Method 2), the first tone is the tone that is being played at the time the first instruction is generated among the plurality of tones. During the playback of the first tone, after the first instruction is generated, the playback of the first tone continues until the end of the first tone represented by the sound data. In the above method, after the first instruction is generated, the playback of the first tone continues until the end of the first tone represented by the sound data. Therefore, compared with the structure that stops the playback of the first tone at the time of the first instruction, the first tone can be appropriately continued in accordance with the sound data.

[0143] In a specific example of Method 1 (Method 3), the multiple notes constitute a musical piece. The moment the user is playing the musical piece is estimated in parallel with the playback of the multiple notes. During the playback of the multiple notes, the playback of the multiple notes follows the performance of the musical piece, corresponding to the estimated result. In the above methods, the playback of the multiple notes follows the performance of the musical piece by the user, thus appropriately reflecting the user's intention (e.g., performance style) or preferences in the playback of the multiple notes.

[0144] In a specific example (method 4) of any of methods 1 to 3, the tone data is performance data that specifies the duration of each of the plurality of tones. During the playback of the first tone, after the first instruction is generated, the playback of the first tone continues until the end of the duration of the duration specified by the performance data for the first tone. In the above methods, after the first instruction is generated, the playback of the first tone continues until the end of the duration of the duration specified by the performance data for the first tone. Therefore, compared to a structure where the playback of the first tone is stopped at the moment of the first instruction, the first tone can be appropriately continued throughout the duration of the duration specified by the performance data.

[0145] In a specific example (method 5) of any of methods 1 to 4, the first instruction and the second instruction are generated in response to an operation performed by the user on the operating device. According to the above method, the user operates the operating device, thereby changing the interval between the first and second tones to an appropriate duration corresponding to the user's intention or preference.

[0146] In a specific example of Method 5 (Method 6), the first indication is generated by the user changing the operating device from a first state to a second state, and the second indication is generated by the user changing the operating device from the second state to the first state. According to the above method, the first indication is generated by the user changing the operating device from a first state to a second state, and after maintaining the second state, the second indication is generated by changing the operating device from the second state to the first state at the user's desired time. Therefore, compared to a structure where the operating device needs to be changed from a first state to a second state for both the first and second indications, the operation of the operating device by the user is simplified.

[0147] In a specific example of Method 5 or Method 6 (Method 7), the duration of the first tone is controlled in accordance with the operating speed of the operating device. According to the above method, the duration of the first tone can be adjusted by the user in accordance with the operating speed of the operating device. Furthermore, the operating device for assigning the first and second instructions also serves to adjust the duration of the first tone, which, compared to a structure where the user operates a separate device for assigning the first and second instructions and adjusting the duration of the first tone, offers the advantage of simplified operation by the user.

[0148] In a specific example (method 8) of any of methods 5 to 7, the timing of starting the playback of the second tone is controlled in accordance with the operating speed of the operating device. According to the above method, the starting point of the second tone playback can be adjusted by the user in accordance with the operating speed of the operating device. Furthermore, the operating device for assigning the first and second instructions also serves to adjust the starting point of the second tone; therefore, compared to a structure where the user operates a separate device for assigning the first and second instructions and adjusting the starting point of the second tone, there is the advantage of simplified operation by the user.

[0149] One aspect (aspect 9) of the present invention relates to a playback control method that plays multiple sounds using sound data representing a time sequence of multiple sounds, such that the first sound playing at the time of the first indication continues until the end of the first sound represented by the sound data. In this aspect, after the first indication is generated, the playback of the first sound continues until the end of the first sound represented by the sound data. Therefore, compared to a structure that stops the playback of the first sound at the time of the first indication, the first sound can be appropriately continued in accordance with the sound data.

[0150] One aspect (aspect 10) of the present invention relates to a playback control system having a playback control unit that plays multiple sounds using sound data representing a time sequence of multiple sounds. The playback control unit plays a first sound corresponding to a first instruction among the multiple sounds, and after the playback of the first sound stops, it starts playing a second sound immediately following the first sound, triggered by a second instruction generated by the user. According to this method, after the playback of the first sound corresponding to the first instruction stops, the playback of the second sound immediately following the first sound begins, triggered by a second instruction from the user. Therefore, the interval (e.g., the duration of a rest) between the first and second sounds can be appropriately controlled according to the times of the first and second instructions.

[0151] One aspect of the present invention (aspect 11) involves a program that causes a computer to function as a playback control unit. This playback control unit plays sound data representing a time sequence of multiple sounds. It plays a first sound corresponding to a first instruction among the multiple sounds, and after the playback of the first sound stops, it begins playback of a second sound immediately following the first sound, triggered by a second instruction generated by the user. According to this method, after the playback of the first sound corresponding to the first instruction stops, playback of the second sound immediately following the first sound begins, triggered by a second instruction from the user. Therefore, the interval between the first and second sounds (e.g., the duration of a rest) can be appropriately controlled according to the timing of the first and second instructions.

[0152] Furthermore, in estimating the moment (performance position) at which the user is playing a piece of music, reference data is needed to compare it with the performance performed by the user. However, it is also conceivable that there may be a situation where no reference data is prepared for the desired piece of music.

[0153] Considering the above, a playback control system according to one aspect (aspect 12) of the present invention comprises: a first recording unit that acquires performance data specifying a plurality of notes indicated to a playback device; a second recording unit that acquires a first sound signal representing a musical note produced by an instrument during a first performance of the piece; a reference data generation unit that generates reference data based on the first sound signal, the reference data representing at least the time of articulation of each musical note involved in the first performance; a performance analysis unit that compares a second sound signal representing a musical note produced by an instrument during a second performance of the piece with the reference data, thereby estimating the time of performance in the piece in parallel with the second performance; and a playback control unit that plays the plurality of notes specified by the performance data, and, in accordance with the result of the estimation by the performance analysis unit, causes the playback of the plurality of notes to follow the second performance. Based on the above structure, reference data representing the sounding times of each musical note involved in the performance is generated from the first acoustic signal representing the musical tone emitted by the instrument during the performance of a piece of music. Therefore, it is not necessary to prepare reference data in advance for the desired piece of music.

[0154] In a specific example of method 12 (method 13), the playback control unit causes the first tone, which is playing at the time the first instruction is generated, to continue until the end of the first tone as indicated by the sound data. After the playback of the first tone stops, the second tone, which immediately follows the first tone, is started to play based on the second instruction from the user. According to the above method, after the playback of the first tone corresponding to the first instruction stops, the second tone, which immediately follows the first tone, is started to play based on the second instruction from the user. Therefore, the interval between the first tone and the second tone (e.g., the duration of a rest) can be appropriately controlled according to each time of the first and second instructions.

[0155] Explanation of the label

[0156] 100…Playback system, 200…Musical instrument, 10…Playback control system, 11…Control device, 12…Storage device, 13…Pickup device, 14…Operating device, 141…Modible part, 20…Playback device, 21…Drive mechanism, 22…Sound-producing mechanism, 50…Performing device, 31…Performing analysis unit, 32…Playback control unit, 33…Instruction receiving unit, 34…Editing processing unit, 41…First recording unit, 42…Second recording unit, 43…Reference data generation unit.

Claims

1. A playback control method, which plays the multiple sounds using sound data representing a time sequence of multiple sounds. In this playback control method, The first tone, which is playing at the moment of the first instruction from the user, continues until the end of the first tone as indicated by the tone data. The first instruction is an instruction to temporarily stop playback. After the first tone stops playing, until a second instruction from the user is received to resume playback stopped by the first instruction, the second tone immediately following the first tone will not be played. Playback of the second tone will begin upon the second instruction from the user. The first and second instructions are generated in accordance with the operations performed by the user on a pedal-type MIDI controller. The sound data refers to the performance data specified for each of the multiple sounds to be played during its production. During the playback of the first note, the playback of the first note continues until the end of the articulation period specified by the performance data with respect to the first note.

2. The playback control method according to claim 1, wherein, The sound data includes musical data that specifies the time sequence of multiple notes that constitute a musical piece. The music data includes: reference data specifying the time sequence of notes of multiple notes played by the user; and performance data specifying the time sequence of notes of multiple notes to be played.

3. The playback control method according to claim 1, wherein, The first instruction is generated by the user switching the pedal-type MIDI controller from state 1 to state 2. The second instruction is generated by the user switching the pedal-type MIDI controller from the second state to the first state.

4. The playback control method according to claim 1, wherein, The duration of the first note is controlled in accordance with the speed of operation of the pedal-type MIDI controller.

5. The playback control method according to claim 1, wherein, The timing of starting the playback of the second note is controlled in accordance with the speed of operation of the pedal-type MIDI controller.

6. A playback control method, which plays the multiple notes using sound data representing a time sequence of multiple notes constituting a musical piece. In this playback control method, The moment when the user is playing the music is presumed in parallel with the playback of the multiple notes. Corresponding to the presumed result, the playback of the multiple notes is made to follow the performance of the music. During the playback of the multiple sounds, The playback of the first tone, which is playing at the moment of the first instruction from the user, is stopped. This first instruction is an instruction to temporarily stop playback. After the first tone stops playing, until a second instruction from the user is received to resume playback stopped by the first instruction, the second tone immediately following the first tone will not be played. Playback of the second tone will begin upon the user's second instruction. The first and second instructions are generated in accordance with the operations performed by the user on a pedal-type MIDI controller. The sound data refers to the performance data specified for each of the multiple sounds to be played during its production. During the playback of the first note, the playback of the first note continues until the end of the articulation period specified by the performance data with respect to the first note.

7. A playback control method, which acquires sound data representing a time sequence of multiple sounds, In this playback control method, The first tone, which is playing at the moment of the first instruction from the user, continues until the end of the first tone as indicated by the tone data. The first instruction is an instruction to temporarily stop playback. After the first tone stops playing, until a second instruction from the user is received to resume playback stopped by the first instruction, the second tone immediately following the first tone will not be played. Playback of the second tone will then begin upon the second instruction from the user. The first and second instructions are generated in accordance with the operations performed by the user on a pedal-type MIDI controller. The sound data refers to the performance data specified for each of the multiple sounds to be played during its production. During the playback of the first note, the playback of the first note continues until the end of the articulation period specified by the performance data with respect to the first note.

8. A playback control system, comprising: The first recording unit acquires performance data for multiple notes specified by the playing device; The second recording unit acquires the first audio signal, which represents the musical tone produced by the instrument during the first performance of a piece of music using an instrument. The reference data generation unit generates reference data based on the first audio signal, which at least represents the sounding time of each musical note involved in the first performance; The performance analysis unit compares the second acoustic signal, representing the musical tone emitted by the instrument during the second performance of the piece, with the reference data, thereby estimating the moment being played within the piece in parallel with the second performance; and The playback control unit plays multiple notes specified by the performance data, and, in accordance with the estimation result made by the performance analysis unit, makes the playback of the multiple notes follow the second performance. The playback control unit causes the first note, which is playing at the moment of the first instruction from the user, to continue until the end of the first note specified by the performance data, whereby the first instruction is an instruction to temporarily stop playback. After the first tone stops playing, until a second instruction from the user is received to resume playback stopped by the first instruction, the second tone immediately following the first tone will not be played. Playback of the second tone will begin upon the user's second instruction. The first and second instructions are generated in accordance with the operations performed by the user on a pedal-type MIDI controller. The performance data specifies the duration of each of the multiple notes to be played. During the playback of the first note, the playback of the first note continues until the end of the articulation period specified by the performance data with respect to the first note.

9. A playback control device that plays the multiple sounds using sound data representing a time sequence of multiple sounds. The device includes a playback control unit that continues playing the first tone among the plurality of tones at the moment a first instruction from the user is generated until the end of the first tone as indicated by the tone data, wherein the first instruction is an instruction to temporarily stop playback. After the first tone stops playing, until a second instruction from the user is received to resume playback stopped by the first instruction, the second tone immediately following the first tone will not be played. Playback of the second tone will begin upon the user's second instruction. The first and second instructions are generated in accordance with the operations performed by the user on a pedal-type MIDI controller. The sound data refers to the performance data specified for each of the multiple sounds to be played during its production. During the playback of the first note, the playback of the first note continues until the end of the articulation period specified by the performance data with respect to the first note.

10. The playback control device according to claim 9, wherein, The sound data includes musical data that specifies the time sequence of multiple notes that constitute a musical piece. The music data includes: reference data specifying the time sequence of notes of multiple notes played by the user; and performance data specifying the time sequence of notes of multiple notes to be played.

11. The playback control device according to claim 9, wherein, The first instruction is generated by the user switching the pedal-type MIDI controller from state 1 to state 2. The second instruction is generated by the user switching the pedal-type MIDI controller from the second state to the first state.

12. The playback control device according to claim 9, wherein, The duration of the first note is controlled in accordance with the speed of operation of the pedal-type MIDI controller.

13. The playback control device according to claim 9, wherein, The timing of starting the playback of the second note is controlled in accordance with the speed of operation of the pedal-type MIDI controller.

14. A playback control device that plays said plurality of notes using sound data representing a time sequence of notes constituting a musical piece. The playback control device has: The performance analysis section, which, in parallel with the playback of the multiple notes, estimates the moment the user is playing within the musical piece; and The playback control unit, in accordance with the result estimated by the performance analysis unit, causes the playback of the multiple notes to follow the performance of the music. The playback control unit during the playback of the multiple sounds The first tone among the plurality of tones that is playing at the moment of the first instruction from the user is executed is stopped from playing, wherein the first instruction is an instruction to temporarily stop playback. After the first tone stops playing, until a second instruction from the user is received to resume playback stopped by the first instruction, the second tone immediately following the first tone will not be played. Playback of the second tone will begin upon the user's second instruction. The first and second instructions are generated in accordance with the operations performed by the user on a pedal-type MIDI controller. The sound data refers to the performance data specified for each of the multiple sounds to be played during its production. During the playback of the first note, the playback of the first note continues until the end of the articulation period specified by the performance data with respect to the first note.

Images