Singing sound output system and method, and musical instrument

Abstract

To output singing sound corresponding to timing and strength of performance input.SOLUTION: A singing sound output system capable of outputting singing sound corresponding to timing and strength of performance input is provided. The singing sound output system includes: an acquisition part 42 for acquiring a series of sound information N including at least information indicating timing and information indicating velocity; a phrase generation part 47 for analyzing accents of the series of sound information N from velocity of individual sound information in the series of acquired sound information N and generating a phrase formed of a plurality of syllables corresponding to the series of sound information N based on the accents; a synthesis part 45 for synthesizing singing sound based on the syllables of the generated phrase; and an output part 45 for outputting the synthesized singing sound.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to a singing sound output system that outputs singing sounds. [Background technology]

[0002] There are known techniques for generating singing sounds in response to musical performance operations. For example, a singing sound synthesis device disclosed in Patent Document 1 generates singing sounds by automatically advancing lyrics character by character or syllable by syllable in response to real-time musical performance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-206323 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, Patent Document 1 does not disclose outputting singing sounds in real time together with accompaniment. If singing sounds are output in real time together with accompaniment, it is difficult to accurately generate singing sounds at the timing that is originally desired. For example, even if a performance operation is started at the timing that is desired to generate, the start of actual singing is delayed because it takes processing time to synthesize and generate singing sounds. Therefore, there is room for improvement in outputting singing sounds at the intended timing together with accompaniment.

[0005] An object of the present invention is to provide a singing sound output system that can output singing sounds according to the timing and strength of a performance input. [Means for solving the problem]

[0006] According to one form of the present invention, a singing sound output system is provided, comprising: an acquisition unit that acquires a series of sound information including at least information indicating timing and information indicating velocity; a phrase generation unit that analyzes the accent of the series of sound information from the velocity of each sound information in the series of sound information acquired by the acquisition unit and generates a phrase consisting of a plurality of syllables corresponding to the series of sound information based on the accent; a synthesis unit that synthesizes a singing sound based on the syllables of the phrase generated by the phrase generation unit; and an output unit that outputs the singing sound synthesized by the synthesis unit. Effect of the Invention

[0007] According to one aspect of the present invention, it is possible to output singing sounds according to the timing and strength of musical performance input. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram showing an overall configuration of a singing sound output system according to a first embodiment. [Diagram 2] FIG. 1 is a block diagram of a singing sound output system. [Diagram 3] FIG. 2 is a functional block diagram of the singing sound output system. [Figure 4] 11 is a timing chart of a process for outputting a singing sound by playing. [Diagram 5] 13 is a flowchart showing system processing. [Figure 6] 11 is a timing chart of a process for outputting a singing sound by playing. [Figure 7] 13 is a flowchart showing system processing. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0010] (First embodiment) 1 is a diagram showing the overall configuration of a singing sound output system according to a first embodiment of the present invention. This singing sound output system 1000 includes a PC (personal computer) 101, a cloud server 102, and a sound output device 103. The PC 101 and the sound output device 103 are communicably connected to the cloud server 102 via a communication network 104 such as the Internet. In the environment in which the PC 101 is used, there are a keyboard 105, a wind instrument 106, and a drum 107 as items or devices for inputting sounds.

[0011] The keyboard 105 and the drums 107 are electronic instruments used to input MIDI (Musical Instrument Digital Interface) signals. The wind instrument 106 is an acoustic instrument used to input monophonic analog sounds. The keyboard 105 and the wind instrument 106 can also input pitch information. The wind instrument 106 may be an electronic instrument, and the keyboard 105 and the drums 107 may be acoustic instruments. These instruments are examples of devices for inputting sound information, and are played by a user on the PC 101 side. The voice of the user on the PC 101 side may also be used as a means for inputting analog sounds, in which case the real voice is input as analog sounds. Therefore, the concept of "playing" for inputting sound information in this embodiment includes inputting the real voice. The device for inputting sound information does not have to be in the form of a musical instrument.

[0012] Although the details will be described later, a typical process by the singing sound output system 1000 will be outlined. A user on the PC 101 side plays an instrument while listening to accompaniment. The PC 101 transmits singing data 51, timing information 52, and accompaniment data 53 (all of which will be described later in FIG. 3) to the cloud server 102. The cloud server 102 synthesizes a singing sound based on the sound generated by the performance of the user on the PC 101 side. The cloud server 102 transmits the singing sound, timing information 52, and accompaniment data 53 to the sound output device 103. The sound output device 103 is a device equipped with a speaker function. The sound output device 103 outputs the received singing sound and accompaniment data 53. At that time, the sound output device 103 outputs the singing sound and the accompaniment data 53 in synchronization with each other based on the timing information 52. The form of "output" here is not limited to playback, but also includes transmission to an external device and recording on a recording medium.

[0013] 2 is a block diagram of the singing sound output system 1000. The PC 101 has a CPU 11, a ROM 12, a RAM 13, a storage unit 14, a timer 15, an operation unit 16, a display unit 17, a sound generating unit 18, an input unit 8, and various I / Fs (interfaces) 19. These components are connected to each other via a bus 10.

[0014] The CPU 11 controls the entire PC 101. The ROM 12 stores various data as well as programs executed by the CPU 11. The RAM 13 provides a work area for the CPU 11 to execute programs. The RAM 13 temporarily stores various information. The storage unit 14 includes a non-volatile memory. The timer 15 measures time. The timer 15 may be a counter type. The operation unit 16 includes a plurality of operators for inputting various information, and accepts instructions from the user. The display unit 17 displays various information. The sound generation unit 18 includes a sound source circuit, an effect circuit, and a sound system.

[0015] The input unit 8 includes an interface for acquiring sound information from devices that input electronic sound information, such as a keyboard 105 or drums 107. The input unit 8 also includes devices such as a microphone for acquiring sound information from devices that input acoustic sound information, such as a wind instrument 106. The various I / Fs 19 are connected to the communication network 104 (FIG. 1) wirelessly or via wires.

[0016] The cloud server 102 has a CPU 21, a ROM 22, a RAM 23, a storage unit 24, a timer 25, an operation unit 26, a display unit 27, a sound generating unit 28, and various I / Fs 29. These components are connected to each other via a bus 20. The configurations of these components are similar to those denoted by the reference numerals 11 to 17 and 19 in the PC 101.

[0017] The sound output device 103 has a CPU 31, a ROM 32, a RAM 33, a storage unit 34, a timer 35, an operation unit 36, a display unit 37, a sound generating unit 38, and various I / Fs 39. These components are connected to each other via a bus 30. The configurations of these components are similar to those denoted by reference numerals 11 to 19 in the PC 101.

[0018] 3 is a functional block diagram of the singing sound output system 1000. The singing sound output system 1000 has a functional block 110. The functional block 110 includes, as individual functional units, a teaching unit 41, an acquisition unit 42, a syllable identification unit 43, a timing identification unit 44, a synthesis unit 45, an output unit 46, and a phrase generation unit 47.

[0019] In this embodiment, as an example, each function of the teaching unit 41 and the acquisition unit 42 is realized by the PC 101. Each of these functions is realized in software by a program stored in the ROM 12. That is, the CPU 11 loads the necessary program in the RAM 13, executes it, and controls various calculations and each hardware resource to provide each function. In other words, these functions are mainly realized by cooperation of the CPU 11, ROM 12, RAM 13, timer 15, display unit 17, sound generation unit 18, input unit 8, and various I / Fs 19. The programs executed here include sequence software.

[0020] Moreover, each function of the syllable identification unit 43, the timing identification unit 44, the synthesis unit 45, and the phrase generation unit 47 is realized by the cloud server 102. Each of these functions is realized in the form of software by a program stored in the ROM 22. These functions are realized mainly by cooperation between the CPU 21, the ROM 22, the RAM 23, the timer 25, and various I / Fs 29.

[0021] The function of the output unit 46 is realized by the sound output device 103. The function of the output unit 46 is realized in software by a program stored in the ROM 32. These functions are realized mainly by cooperation of the CPU 31, ROM 32, RAM 33, timer 35, sound generating unit 38, and various I / Fs 39.

[0022] The singing sound output system 1000 refers to singing data 51, timing information 52, accompaniment data 53 and a phrase database 54. The phrase database 54 is stored in advance in, for example, the ROM 12. Note that the phrase generating unit 47 and the phrase database 54 are not essential to this embodiment. These will be described in the third embodiment later.

[0023] Singing data 51, timing information 52, and accompaniment data 53 are associated with each other for each song and are stored in advance in ROM 12. Accompaniment data 53 is information for playing the accompaniment of each song recorded as sequence data. Singing data 51 includes a plurality of syllables. Singing data 51 includes lyrics text data and a phonological information database. The lyrics text data is data describing lyrics, and lyrics for each song are described divided into syllables. In each song, the accompaniment positions of accompaniment data 53 and syllables in singing data 51 are associated in time by timing information 52.

[0024] The processing by each functional unit in the functional block 110 will be described in detail with reference to Figs. 4 and 5. Here, an outline will be described. The teaching unit 41 indicates (instructs) the user of the progress position in the singing data 51. The acquisition unit 42 acquires at least one sound information N (see Fig. 4) input by playing. The syllable identification unit 43 identifies a syllable corresponding to the acquired sound information N from the multiple syllables in the singing data 51. The timing identification unit 44 associates the sound information N with a difference ΔT (see Fig. 4) as relative information indicating a relative timing with respect to the identified syllable. The synthesis unit 45 synthesizes a singing sound based on the identified syllable. The output unit 46 outputs the synthesized singing sound in synchronization with the accompaniment sound based on the accompaniment data 53 based on the relative information.

[0025] FIG. 4 is a timing chart of the process of outputting singing sounds by playing. When a song is selected and the process is started, as shown in FIG. 4, the syllables corresponding to the progression position in the singing data 51 are displayed to the user on the PC 101. For example, the syllables are displayed in order, such as "sa", "ku", and "ra". The pronunciation start timing t (t1 to t3) is determined by the time correspondence with the accompaniment data 53, and is the pronunciation start timing of the original syllable determined in the singing data 51. For example, the time t1 indicates the pronunciation start position of the syllable "sa" on the singing data 51. In parallel with the instruction of the progression of syllables, the accompaniment based on the accompaniment data 53 also progresses.

[0026] The user plays along with the progression of the indicated syllables. Here, an example is given in which a MIDI signal is input by playing on a keyboard 105 capable of inputting pitch information. The user, who is the performer, presses the keys corresponding to each syllable in sequence in time with the start timing of each of the syllables "sa," "ku," and "ra." In this way, sound information N (N1 to N3) is obtained sequentially. The pronunciation length of each piece of sound information N is the time from the input start timing s (s1 to s3) to the input end timing e (e1 to e3). The input start timing s corresponds to note-on, and the input end timing e corresponds to note-off. The sound information N includes pitch information and velocity.

[0027] A user may purposely shift the actual input start timing s from the sound production start timing t. In the cloud server 102, the shift time of the input start timing s from the sound production start timing t is calculated as a time difference ΔT (ΔT1 to T3) (relative information). The difference ΔT is calculated for each syllable and associated with each syllable. The cloud server 102 synthesizes a singing sound based on the sound information N, and sends it to the sound output device 103 together with the accompaniment data 53.

[0028] The sound output device 103 outputs the singing sound and the accompaniment sound based on the accompaniment data 53 in synchronization with each other. At this time, the sound output device 103 outputs the accompaniment sound at a set constant tempo. The sound output device 103 outputs the singing sound while matching each syllable with the accompaniment position based on the timing information 52. Note that a processing time is required from the input of the sound information N to the output of the singing sound. Therefore, the sound output device 103 uses a delay process to delay the output of the accompaniment sound in order to match each syllable with the accompaniment position.

[0029] For example, the sound output device 103 adjusts the output timing by referring to the difference ΔT corresponding to each syllable. As a result, the singing sound starts to be output exactly according to the input timing (at the input start timing s). For example, the output (pronunciation) of the syllable "ku" starts at a timing earlier than the pronunciation start timing t2 by the difference ΔT2. Also, the output (pronunciation) of the syllable "ra" starts at a timing later than the pronunciation start timing t3 by the difference ΔT3. The pronunciation of each syllable ends (is muted) at the time corresponding to the input end timing e. Therefore, the accompaniment sound is output at a fixed tempo, and the singing sound is output at a timing according to the performance timing. Therefore, the singing sound can be output at the timing when the sound information N is input, in synchronization with the accompaniment.

[0030] 5 is a flowchart showing a system process for outputting a singing sound by playing the singing sound output system 1000. In this system process, a PC process executed by the PC 101, a cloud server process executed by the cloud server 102, and a sound output device process executed by the sound output device 103 are executed in parallel. The PC process is realized by the CPU 11 expanding a program stored in the ROM 12 into the RAM 13 and executing it. The cloud server process is realized by the CPU 21 expanding a program stored in the ROM 22 into the RAM 23 and executing it. The sound output device process is realized by the CPU 31 expanding a program stored in the ROM 32 into the RAM 33 and executing it. Each of these processes is started when the start of the system process is instructed in the PC 101.

[0031] First, the PC process will be described. In step S101, the CPU 11 of the PC 101 selects a song to be played this time (hereinafter, referred to as a selected song) from among a plurality of songs prepared based on an instruction from the user. The playing tempo of each song is preset as a default for each song. However, when a song to be played is selected, the CPU 11 may change the tempo to be set based on an instruction from the user.

[0032] In step S102, the CPU 11 transmits the related data (the singing data 51, the timing information 52, and the accompaniment data 53) corresponding to the selected song to the cloud server 102 through the various I / Fs 19.

[0033] In step S103, the CPU 11 starts teaching the moving position. Accordingly, the CPU 11 transmits a notification that teaching of the moving position has started to the cloud server 102. The teaching process here is realized, for example, by executing sequence software. The CPU 11 (teaching unit 41) uses the timing information 52 to teach the current moving position.

[0034] For example, the display unit 17 displays lyrics corresponding to syllables in the singing data 51. The CPU 11 teaches the progression position on the displayed lyrics. For example, the teaching unit 41 indicates the progression position by changing the display mode, such as color, of the lyrics at the current position, or by moving the cursor position or the position of the lyrics themselves. Furthermore, the CPU 11 indicates the progression position by playing the accompaniment data 53 at a set tempo. Note that the method of indicating the progression position is not limited to these exemplified modes, and various methods that allow recognition by sight or hearing can be adopted. For example, a method of indicating the note at the current position on a displayed musical score may be used. Alternatively, a metronome sound may be generated after indicating the start timing. Also, at least one method may be adopted, and multiple methods may be combined.

[0035] In step S104, the CPU 11 (acquisition unit 42) executes a sound information acquisition process. For example, the user plays in time with the lyrics while checking the progression position that he or she has been taught (for example, while listening to the accompaniment). The CPU 11 acquires MIDI data or analog sound from the performance as sound information N. The sound information N normally includes information on input start timing s, input end timing e, pitch information, and velocity. Note that, as in the case where the drum 107 is played, pitch information is not necessarily included. Velocity information may be canceled. The input start timing s and input end timing e are defined by relative times with respect to the accompaniment progression. Note that, when analog sound such as a human voice is acquired by a microphone, audio data is acquired as the sound information N.

[0036] In step S105, CPU 11 transmits the sound information N acquired in step S104 to cloud server 102. In step S106, it is determined whether the selected song has ended, that is, whether teaching of the progress position up to the last position in the selected song has been completed. If the selected song has not ended, CPU 11 returns to step S104. Therefore, until the selected song ends, sound information N acquired in accordance with the performance as the song progresses is transmitted to cloud server 102 at any time. When the selected song ends, CPU 11 transmits a notification to that effect to cloud server 102 and terminates PC processing.

[0037] Next, the cloud server process will be described. In step S201, the CPU 21 of the cloud server 102 receives related data corresponding to the selected song through the various I / Fs 29, and proceeds to step S202. In step S202, the CPU 21 transmits the received related data to the sound output device 103 through the various I / Fs 29. Note that it is not necessary to transmit the singing data 51 to the sound output device 103.

[0038] In step S203, the CPU 21 starts a series of processes (S204 to S209). At the start of the series of processes, the CPU 21 executes sequence software and advances time using the received associated data while waiting for reception of the next sound information N. In step S204, the CPU 21 receives the sound information N.

[0039] In step S205, the CPU 21 (the syllable identification unit 43) identifies a syllable corresponding to the received sound information N. First, the CPU 21 calculates, for each syllable, a difference ΔT between an input start timing s in the sound information N and a pronunciation start timing t for each of the multiple syllables in the singing data 51 corresponding to the selected song. Then, the CPU 21 identifies, among the multiple syllables in the singing data 51, the syllable with the smallest difference ΔT as the syllable corresponding to the currently received sound information N.

[0040] 4, for the sound information N2, the difference ΔT2 between the input start timing s2 and the pronunciation start timing t2 of the syllable "ku" is the smallest compared to the differences with other syllables. Therefore, the CPU 21 specifies the syllable "ku" as the syllable corresponding to the sound information N2. In this way, for each piece of sound information N, the syllable corresponding to the pronunciation start timing t closest to the input start timing s is specified as the corresponding syllable.

[0041] When the sound information N is audio data, the CPU 21 (syllable specification unit 43) determines the on / off timing, pitch, and velocity of the sound information N by analysis.

[0042] In step S206, the CPU 21 (timing specification unit 44) executes a timing specification process. That is, the CPU 21 associates the currently received sound information N with the syllable specified as the syllable corresponding to the sound information N with a difference ΔT.

[0043] In step S207, the CPU 21 (synthesizer 45) synthesizes a singing sound based on the specified syllable. The pitch of the singing sound is determined by the pitch information of the corresponding sound information N. If the sound information N is a drum sound, the pitch of the singing sound may be, for example, a constant pitch. The output timing of the singing sound is determined by the sound start timing t and the input end timing e (or sound length) of the corresponding sound information N. Thus, the singing sound is synthesized from the syllable corresponding to the sound information N at a pitch determined by the performance. Note that, because the sound is muted too late during the performance, the sound period of the current syllable may overlap with the original sound timing of the next syllable on the singing data. In this case, the input end timing e may be corrected so that the sound is forcibly muted before the original sound timing of the next syllable.

[0044] In step S208, the CPU 21 executes data transmission. That is, the CPU 21 transmits the synthesized singing sound, the difference ΔT corresponding to the syllable, and the velocity information at the time of performance to the sound output device 103 through the various I / Fs 29.

[0045] In step S209, the CPU 21 determines whether the selected song has ended, that is, whether a notification that the selected song has ended has been received from the PC 101. If the selected song has not ended, the CPU 21 returns to step S204. Therefore, until the selected song ends, singing sounds based on syllables corresponding to the sound information N are synthesized and transmitted at any time. The CPU 21 may determine that the selected song has ended when a predetermined time has passed since the processing of the last received sound information N data ended. When the selected song ends, the CPU 21 ends the cloud server processing.

[0046] Next, the sound output device process will be described. In step S301, the CPU 31 of the sound output device 103 receives related data corresponding to the selected song through the various I / Fs 39, and proceeds to step S302. In step S302, the CPU 31 receives the data (singing sound, difference ΔT, velocity) transmitted from the cloud server 102 in step S208.

[0047] In step S303, the CPU 31 (the output unit 46) executes synchronous output of the singing sound and the accompaniment based on the received singing sound and the difference ΔT, the already received accompaniment data 53, and the timing information 52.

[0048] As explained in Fig. 4, the CPU 31 outputs the accompaniment sound based on the accompaniment data 53, and in parallel with this, outputs the singing sound while adjusting the output timing based on the timing information and the difference ΔT. Here, playback is adopted as a representative mode of synchronous output of the accompaniment sound and the singing sound. Therefore, the sound output device 103 allows the user of the PC 101 to listen to the performance in synchronization with the accompaniment.

[0049] The mode of synchronous output is not limited to playback, and may be recorded in the storage unit 34 as an audio file, or may be transmitted to an external device via the various I / Fs 39.

[0050] In step S304, the CPU 31 determines whether the selected song has ended, that is, whether a notification that the selected song has ended has been received from the cloud server 102. If the selected song has not ended, the CPU 31 returns to step S302. Therefore, the synchronous output of the received singing sound continues until the selected song ends. The CPU 31 may determine that the selected song has ended when a predetermined time has elapsed since the processing of the last received data ended. When the selected song ends, the CPU 31 ends the sound output device processing.

[0051] According to this embodiment, a syllable corresponding to the acquired sound information N while indicating the progression position in the singing data 51 to the user is identified from a plurality of syllables in the singing data 51. Relative information (difference ΔT) is associated with the sound information N, and a singing sound is synthesized based on the identified syllable. Based on the relative information, the singing sound and the accompaniment sound based on the accompaniment data 53 are output in synchronization with each other. Therefore, the singing sound can be output in synchronization with the accompaniment at the timing when the sound information N is input.

[0052] Furthermore, if the sound information N includes pitch information, the singing sound can be output at the pitch input by playing. Furthermore, if the sound information N includes velocity information, the singing sound can be output at a volume according to the strength of playing.

[0053] Note that the related data (vocal data 51, timing information 52, accompaniment data 53) is transmitted to cloud server 102 and sound output device 103 after the selected song is determined, but is not limited to this. For example, related data for a plurality of songs may be stored in advance in cloud server 102 and sound output device 103. Then, when the selected song is determined, information identifying the selected song may be transmitted to cloud server 102 and further to sound output device 103.

[0054] (Second embodiment) In the second embodiment of the present invention, some system processing is different from that in the first embodiment. Therefore, the differences from the first embodiment will be mainly described with reference to Figures 5 and 6. In the first embodiment, the performance tempo was fixed, but in this embodiment, the performance tempo is variable and changes depending on the performance by the performer.

[0055] FIG. 6 is a timing chart of the process of outputting singing sounds by playing. The order of multiple syllables in the singing data 51 is predetermined. In FIG. 6, in the syllable progression display, the singing sound output system 1000 shows the next syllable in the singing data to the user while waiting for the input of sound information N, and advances the syllable indicating the progression position to the next syllable by one each time sound information N is input. Therefore, the syllable progression display waits until there is a performance input corresponding to the next syllable. The progression instruction of the accompaniment data also waits until there is a performance input in accordance with the syllable progression.

[0056] The cloud server 102 identifies the next syllable in the progression at the time when the sound information N is input as the syllable corresponding to the input sound information N. Therefore, each time a key is pressed, a corresponding syllable is identified in order.

[0057] The actual input start timing s may deviate from the sound production start timing t. As in the first embodiment, in the cloud server 102, the delay time of the input start timing s from the sound production start timing t is calculated as a time difference ΔT (ΔT1 to T3) (relative information). The difference ΔT is calculated for each syllable and associated with each syllable. The cloud server 102 synthesizes a singing sound based on the sound information N, and sends it to the sound output device 103 together with the accompaniment data 53.

[0058] 6, syllable pronunciation start timing t' (t1' to t3') is the pronunciation start timing of the syllable at the time of output. The syllable pronunciation start timing t' is determined by the input start timing s. The progress of the accompaniment sound at the time of output also changes from time to time depending on the syllable pronunciation start timing t'.

[0059] The sound output device 103 outputs the singing sound and the accompaniment sound based on the accompaniment data 53 in synchronization with each other by adjusting the output timing based on the timing information and the difference ΔT. In this case, the sound output device 103 outputs the singing sound at the syllable pronunciation start timing t'. The sound output device 103 outputs the accompaniment sound while matching each syllable with the accompaniment position based on the difference ΔT. In order to match each syllable with the accompaniment position, the sound output device 103 delays the output of the accompaniment sound using a delay process. Therefore, the singing sound is output at a timing according to the performance timing, and the tempo of the accompaniment sound changes according to the performance timing.

[0060] The system processing in this embodiment will be described with reference to the flowchart in Fig. 5. Parts that are not specifically mentioned are the same as those in the first embodiment.

[0061] In the teaching process started in step S103 in the PC 101, the CPU 11 (teaching unit 41) teaches the current progress position using the timing information 52. In step S104, the CPU 11 (acquisition unit 42) executes sound information acquisition process. The user plays and inputs a sound corresponding to the next syllable while checking the progress position. The CPU 11 waits for the teaching progression of the syllable and the accompaniment progression until the next sound information N is input. Therefore, the CPU 11 teaches the next syllable while waiting for the input of the sound information N, and advances the syllable indicating the progress position by one to the next syllable each time the sound information N is input. The CPU 11 also matches the accompaniment progression to the teaching progression of the syllable.

[0062] In the series of processes starting in step S203 in the cloud server 102, the CPU 21 advances time while waiting for reception of sound information N. In step S204, the CPU 21 receives sound information N at any time, and advances time when it receives sound information N. Thus, the CPU 21 waits for the progression of time until the next sound information N is received.

[0063] When the sound information N is received, in step S205, the CPU 21 (the syllable identification unit 43) identifies the syllable corresponding to the received sound information N. Here, the CPU 21 identifies the next syllable in the progression order at the time when the sound information N was input as the syllable corresponding to the currently received sound information N. Therefore, each time a key is turned on by playing, the corresponding syllable is identified in order.

[0064] After identifying the syllable, in step S206, the CPU 21 calculates a difference ΔT and associates it with the identified syllable. That is, as shown in Fig. 6, the CPU 21 obtains a shift time between the pronunciation start timing t corresponding to the identified syllable and the input start timing s as the difference ΔT. Then, the CPU 21 associates the obtained difference ΔT with the identified syllable.

[0065] In the data transmission in step S208, the CPU 21 transmits the synthesized singing sound, the difference ΔT corresponding to the syllable, and the velocity at the time of performance to the sound output device 103 via the various I / Fs 29.

[0066] In the sound output device 103, in the synchronous output process executed in step S303, the CPU 31 (output unit 46) executes synchronous output of the singing sound and the accompaniment based on the received singing sound and difference ΔT, the already received accompaniment data 53, and the timing information 52. At that time, the CPU 31 performs the output process while matching each syllable with the accompaniment position by adjusting the output timing of the accompaniment sound and the singing sound with reference to the difference ΔT.

[0067] As a result, as shown in Fig. 6, the singing sound starts to be output exactly at the input timing (at input start timing s). For example, the output (pronunciation) of the syllable "ku" starts at a timing earlier than the pronunciation start timing t2 by a difference ΔT2. Also, the output (pronunciation) of the syllable "ra" starts at a timing later than the pronunciation start timing t3 by a difference ΔT3. The pronunciation of each syllable ends at the time corresponding to the input end timing e.

[0068] On the other hand, the performance tempo of the accompaniment sounds changes in accordance with the performance timing. For example, the CPU 31 corrects the position of the sound production start timing t2 for the accompaniment sounds to the position of the sound production start timing t2' and outputs the corrected position.

[0069] Therefore, the accompaniment sound is output at a variable tempo, and the singing sound is output at a timing corresponding to the performance timing, so that the singing sound can be output at the timing when the sound information N is input, in synchronization with the accompaniment.

[0070] According to this embodiment, the teaching unit 41 indicates the next syllable while waiting for the input of the sound information N, and advances the syllable indicating the progression position to the next syllable each time the sound information N is input. Then, the syllable identification unit 43 identifies the next syllable in the progression order at the time when the sound information N is input as the syllable corresponding to the input sound information N. Therefore, the same effect as that of the first embodiment can be achieved in terms of outputting the singing sound at the timing when the sound information N is input in synchronization with the accompaniment. Also, even if the user plays at a free tempo, the singing sound can be output in synchronization with the accompaniment according to the user's performance tempo.

[0071] In the first and second embodiments, the relative information associated with the sound information N is not limited to the difference ΔT. For example, the relative information indicating the relative timing for the identified syllable may be the relative time of the sound information N and the relative time of each syllable with respect to a certain time defined by the timing information 52.

[0072] (Third embodiment) A third embodiment of the present invention will be described with reference to Figs. 1 to 3 and 7. If singing sounds can be generated using a device such as a drum that does not allow pitch information to be input, the enjoyment will be expanded. Therefore, in this embodiment, a drum 107 is used for performance input. In this embodiment, when a user freely strikes the drum 107 without being taught the accompaniment or the progression of syllables, a singing phrase is generated for each unit of a series of sound information N acquired by the strike. The basic configuration of the singing sound output system 1000 is the same as that of the first embodiment. In this embodiment, it is assumed that the performance input is performed using the drum 107, and it is assumed that there is no pitch information, so a control different from that of the first embodiment is applied.

[0073] In this embodiment, the teaching unit 41, timing identification unit 44, singing data 51, timing information 52, and accompaniment data 53 shown in Fig. 3 are not essential. The phrase generation unit 47 analyzes the accent of the series of sound information N from the velocity of each piece of sound information N in the series of sound information N, and generates a phrase consisting of a plurality of syllables corresponding to the series of sound information N based on the accent. The phrase generation unit 47 generates a phrase corresponding to the series of sound information N by extracting a phrase matching the accent from a phrase database 54 including a plurality of phrases prepared in advance. A phrase having the number of syllables constituting the series of sound information N is extracted.

[0074] Here, the accent of a series of sound information N refers to a stress accent due to the relative strength of the sounds. The accent of a phrase refers to a pitch accent due to the relative pitch of each syllable. Therefore, the strength of the sounds in the sound information N corresponds to the pitch of the phrase.

[0075] 7 is a flowchart showing a system process for outputting a singing sound by playing the singing sound output system 1000. In this system process, the executing subjects, execution conditions, and start conditions of the PC process, the cloud server process, and the sound output device process are the same as those of the system process shown in FIG.

[0076] First, the PC process will be described. In step S401, the CPU 11 of the PC 101 transitions to a performance start state based on an instruction from the user. At that time, the CPU 11 transmits a notification that the state has transitioned to the performance start state to the cloud server 102 via the various I / Fs 19.

[0077] In step S402, when the user strikes the drum 107, the CPU 11 (acquisition unit 42) acquires corresponding sound information N. The sound information N is MIDI data or analog sound. The sound information N includes at least information indicating the input start timing (strike on) and information indicating the velocity.

[0078] In step S403, the CPU 11 (acquisition unit 42) determines whether the current series of sound information N has been finalized. For example, when the first sound information N is input within a first predetermined time after the transition to the performance start state, the CPU 11 determines that the series of sound information N has been finalized when a second predetermined time has elapsed since the last sound information N was input. The series of sound information N is assumed to be a group of multiple sound information N, but may be a single piece of sound information N.

[0079] In step S404, the CPU 11 transmits the acquired series of sound information N to the cloud server 102. In step S405, the CPU 11 determines whether or not the user has instructed the performance to end. If the user has not instructed the performance to end, the CPU 11 returns to step S402, and if the user has instructed the performance to end, the CPU 11 transmits a notification to that effect to the cloud server 102 and terminates the PC processing. Thus, each time a set of sound information N is determined, the set of sound information N is transmitted.

[0080] Next, the cloud server process will be described. When the CPU 21 receives a notification that the state has shifted to the performance start state, the CPU 21 starts a series of processes (S502 to S506) in step S501. In step S502, the CPU 21 receives the series of sound information N transmitted from the PC 101 in step S404.

[0081] In step S503, the CPU 21 (phrase generating unit 47) generates one phrase for the current series of sound information N. An example of this method is shown below. For example, the CPU 21 analyzes the accent of the series of sound information N from the velocity of each piece of sound information N, and extracts phrases that match the accent and the number of syllables that make up the series of sound information N from the phrase database 54. At this time, the extraction range may be narrowed down depending on conditions. For example, the phrase database 54 may be classified by conditions, and the user may be able to set at least one of the conditions such as "noun," "fruit," "stationery," "color," and "size."

[0082] For example, consider the case where the number of sound information N is four and the condition is "fruit." If the analyzed accent is "strong-weak-weak-weak," "durian" will be extracted, and if the accent is "weak-strong-weak-weak," "orange" will be extracted. Consider the case where the number of sound information N is four and the condition is "stationery." If the analyzed accent is "strong-weak-weak-weak," "compass" will be extracted, and if the accent is "weak-strong-weak-weak," "crayon" will be extracted. Note that it is not mandatory to set conditions.

[0083] In step S504, the CPU 21 (synthesizer 45) synthesizes a singing voice from the generated phrase. The pitch of the singing voice may conform to the pitch of each syllable set in the phrase. In step S505, the CPU 21 transmits the singing voice to the sound output device 103 via the various I / Fs 29.

[0084] In step S506, the CPU 21 determines whether or not a notification that an instruction to end the performance has been received from the PC 101. If the CPU 21 has not received such a notification, the process returns to step S502. If the CPU 21 has received such a notification, the CPU 21 transmits such a notification to the sound output device 103 and ends the cloud server process.

[0085] Next, the sound output device process will be described. In step S601, the CPU 31 of the sound output device 103 receives singing sounds through the various I / Fs 39, and proceeds to step S602. In step S602, the CPU 31 (output unit 46) outputs the received singing sounds. The output timing of each syllable depends on the input timing of the corresponding sound information N. The form of output here is not limited to playback, as in the first embodiment.

[0086] In step S603, it is determined whether or not a notification that an instruction to end the performance has been received from the cloud server 102. If the CPU 31 has not received such a notification, it returns to step S601, and if the CPU 31 has received such a notification, it ends the sound output device processing. Therefore, the CPU 31 outputs the singing sound of the phrase whenever it receives it.

[0087] According to this embodiment, it is possible to output singing sounds that correspond to the timing and strength of musical performance input.

[0088] In this embodiment, since the tone color differs between a strike on the head and a strike on the rim (rim shot), this difference in tone color may be used as a parameter for phrase generation. For example, the above conditions for phrase extraction may be different between a strike on the head and a rim shot.

[0089] Note that the object that produces sound by striking is not limited to a drum, but may be clapping of the hands. If an electronic drum is used, the striking position on the head may be detected, and the difference in striking position may be used as a parameter for generating phrases.

[0090] In this embodiment, if the obtainable sound information N includes pitch information, the high and low pitches may be replaced with accents and processing may be performed in the same way as when striking a drum. For example, when "Do-Mi-Do" is played on the piano, a phrase equivalent to "Weak-Strong-Weak" played on the drums may be extracted.

[0091] In each of the above embodiments, if the sound output device 103 has multiple singing voices (multiple genders, etc.), the singing voice to be used may be switched according to the sound information N. For example, if the sound information N is audio data, the singing voice may be switched according to the tone color. If the sound information N is MIDI data, the singing voice may be switched according to the tone color and other parameters set in the PC 101.

[0092] In the above-mentioned embodiments, it is not essential that the singing sound output system 1000 includes the PC 101, the cloud server 102, and the sound output device 103. It is also not limited to a system that uses a cloud server. That is, each functional unit shown in FIG. 3 may be realized by any device, or may be realized by one device. If the above-mentioned functional units are realized by one integrated device, the device may not be called a singing sound output system, but may be called a singing sound output device.

[0093] In each of the above embodiments, at least a part of each functional unit shown in FIG. 3 may be realized by AI (Artificial Intelligence).

[0094] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0095] In addition, a storage medium storing a control program represented by software for achieving the present invention may be read into the present system to achieve the same effect as the present invention. In that case, the program code itself read from the storage medium realizes the novel function of the present invention, and the non-transient computer-readable storage medium storing the program code constitutes the present invention. The program code may also be supplied via a transmission medium, etc., in which case the program code itself constitutes the present invention. In addition, as the storage medium in these cases, in addition to ROM, a floppy disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a non-volatile memory card, etc. may be used. Non-transient computer-readable storage media also include those that hold a program for a certain period of time, such as a volatile memory (e.g., DRAM (Dynamic Random Access Memory)) inside a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. [Explanation of symbols]

[0096] 42 Acquisition unit, 45 Synthesis unit, 46 Output unit, 47 Phrase generation unit, 1000 Singing sound output system

Claims

1. an acquisition unit that acquires a series of sound information including at least information indicating timing and information indicating velocity; a phrase generating unit that analyzes an accent of the series of sound information from the velocity of each piece of sound information in the series of sound information acquired by the acquiring unit, and generates a phrase consisting of a plurality of syllables corresponding to the series of sound information based on the accent; a synthesis unit that synthesizes singing sounds based on the syllables of the phrases generated by the phrase generation unit; An output unit that outputs the singing sound synthesized by the synthesis unit.

2. The singing sound output system according to claim 1 , wherein the phrase generating unit generates a phrase corresponding to the series of sound information by extracting a phrase that matches the accent from a database of phrases prepared in advance.

Images