Electronic musical instrument, electronic musical instrument control method, and program
The electronic musical instrument infers audio waveforms that match tempo changes by using a pitch specifying section and pronunciation model, improving expressiveness by adapting to different playing speeds.
Patent Information
- Application Number
- JP2023073896
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2023-04-28
- Publication Date
- 2024-11-06
- Estimated Expiration
- 2040-09-11
AI Technical Summary
Existing electronic musical instruments struggle to infer appropriate audio waveforms in real-time that match changes in performance speed between notes, leading to insufficient expressiveness when playing at different tempos.
An electronic musical instrument that includes a pitch specifying section, performance form data conversion, and a pronunciation model to input performance data into a learned acoustic model, synthesizing musical tone data that adapts to changes in tempo and style.
Enables the inference of appropriate audio waveforms that match real-time changes in performance speed, enhancing expressiveness by producing rich and lively sounds at varying tempos.
Smart Images

Figure 0007578156000015 
Figure 0007578156000016 
Figure 0007578156000017
Abstract
Description
[Technical field]
[0001] The present invention relates to an electronic musical instrument that drives a trained acoustic model in response to the operation of a control element such as a keyboard to output sound, a method for controlling an electronic musical instrument, and a program. [Background technology]
[0002] In electronic musical instruments, in order to compensate for the weakness of the expressiveness of conventional PCM (Pulse Code Modulation) methods in terms of the expressiveness of singing voices and acoustic instruments, a technology has been devised and is becoming practical in which an acoustic model that models the human vocal mechanism and the sound generation mechanism of an acoustic instrument using digital signal processing is trained using machine learning based on singing and playing movements, and the trained acoustic model is then driven based on actual playing operations to infer and output audio waveform data of singing voices and musical sounds (for example, Patent Document 1). [Prior art documents] [Patent documents]
[0003] [Patent Document 1] Patent No. 6610714 Summary of the Invention [Problem to be solved by the invention]
[0004] When using machine learning to generate vocal or musical waveforms, the generated waveforms often change depending on the tempo of the performance, the way the phrase is sung, and the performance style. For example, the duration of the consonants in vocals, the duration of the blown sounds in wind instruments, and the duration of the noise components when the strings of a bowed string instrument begin to be rubbed are longer in slow performances with few notes, resulting in an expressive and vivid sound, and shorter in fast performances with many notes, resulting in a crisp sound.
[0005] However, when a user plays in real time on a keyboard or the like, there is no way to communicate to the sound source device the changing playing speed between notes that corresponds to changes in the notation of each note or differences in the played phrases, so the acoustic model cannot infer an appropriate audio waveform that corresponds to changes in the playing speed between notes. This can result in problems such as a lack of expressiveness when playing slowly, or conversely, a delayed rise in the audio waveform generated for fast-tempo playing, making it difficult to play.
[0006] Therefore, an object of the present invention is to make it possible to infer an appropriate voice waveform that matches the change in performance speed between notes that changes in real time. [Means for solving the problem]
[0007] The electronic musical instrument according to one embodiment of the present invention comprises: a pitch designation unit that outputs performance-time pitch data designated during performance; A value of performance tempo data is obtained based on a velocity value when an initial pitch is designated by the performance during the performance, and the performance tempo data is Performance style data indicating the performance style at the time of performance As and a pronunciation model section which synthesizes and outputs musical tone data corresponding to the performance pitch data and the performance style data during performance based on acoustic model parameters inferred by inputting the performance pitch data and the performance style data during performance to a trained acoustic model during performance.
[0008] The electronic musical instrument according to another embodiment of the present invention includes a lyrics output unit that outputs lyrics data during performance that indicates lyrics during performance, and a pitch designation unit that outputs pitch data during performance that is designated in accordance with the output of the lyrics during performance. A value of performance tempo data is obtained based on a velocity value when an initial pitch is designated by the performance during the performance, and the performance tempo data is Performance style data indicating the performance style at the time of performance As and a vocalization model section which synthesizes and outputs singing voice data corresponding to the lyrics data at the time of performance, the pitch data at the time of performance, and the performance style data at the time of performance based on acoustic model parameters inferred by inputting the lyrics data at the time of performance, the pitch data at the time of performance, and the performance style data at the time of performance to a trained acoustic model. Effect of the Invention
[0009] According to the present invention, it is possible to infer an appropriate voice waveform that matches the change in performance speed between notes that changes in real time. [Brief description of the drawings]
[0010] [Figure 1] 1 is a diagram showing an example of the appearance of an embodiment of an electronic keyboard instrument; [Diagram 2] 1 is a block diagram showing an example of a hardware configuration of an embodiment of a control system for an electronic keyboard instrument; [Diagram 3] 3 is a block diagram showing an example of the configuration of a voice learning unit and a voice synthesis unit; FIG. [Figure 4] FIG. 1 is an explanatory diagram showing an example of musical notation that is the basis for singing. [Diagram 5] FIG. 13 is a diagram showing a change in the waveform of a singing voice caused by a difference in performance tempo. [Figure 6] 2 is a block diagram showing an example of the configuration of a lyrics output unit, a pitch designation unit, and a performance style output unit. FIG. [Figure 7] FIG. 2 is a diagram illustrating an example of a data configuration according to the present embodiment. [Figure 8] 4 is a main flowchart showing an example of a control process of the electronic musical instrument according to the present embodiment. [Figure 9] 11 is a flowchart showing a detailed example of an initialization process, a tempo change process, and a song start process. [Figure 10] 13 is a flowchart showing a detailed example of a switch process. [Figure 11] 13 is a flowchart showing a detailed example of keyboard processing. [Figure 12] 13 is a flowchart showing a detailed example of an automatic performance interrupt process. [Figure 13] 10 is a flowchart showing a detailed example of a song reproducing process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
[0012] 1 is a diagram showing an example of the appearance of an embodiment 100 of an electronic keyboard instrument. The electronic keyboard instrument 100 includes a keyboard 101 consisting of a plurality of keys as operators, a first switch panel 102 for instructing various settings such as volume specification, song playback tempo setting described later, performance tempo mode setting described later, performance tempo adjustment setting described later, song playback start described later, and accompaniment playback described later, a second switch panel 103 for selecting songs and accompaniments and selecting tones, and an LCD 104 (Liquid Crystal Display) for displaying lyrics, musical scores, and various setting information during song playback described later. Although not shown in particular, the electronic keyboard instrument 100 also includes a speaker on the back, side, or rear part for emitting musical tones generated by playing the instrument.
[0013] Fig. 2 is a diagram showing an example of a hardware configuration of an embodiment of a control system 200 of the electronic keyboard instrument 100 of Fig. 1. In Fig. 2, the control system 200 includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, a sound source LSI (Large Scale Integrated Circuit) 204, a voice synthesis LSI 205, a key scanner 206 to which the keyboard 101, the first switch panel 102, and the second switch panel 103 of Fig. 1 are connected, an LCD controller 208 to which the LCD 104 of Fig. 1 is connected, and a network interface 219 for exchanging MIDI data with an external network, all of which are connected to a system bus 209. A timer 210 for controlling the sequence of automatic performance is also connected to the CPU 201. Furthermore, the musical sound data 218 and singing voice data 217 outputted from the sound source LSI 204 and the voice synthesis LSI 205, respectively, are converted into an analog musical sound output signal and an analog singing voice output signal by D / A converters 211 and 212. The analog musical sound output signal and the analog singing voice output signal are mixed by a mixer 213, and the mixed signal is amplified by an amplifier 214, and then outputted from a speaker or an output terminal (not shown).
[0014] The CPU 201 executes a control program loaded from the ROM 202 to the RAM 203 while using the RAM 203 as a working memory, thereby executing the control operation of the electronic keyboard instrument 100 shown in Fig. 1. The ROM 202 also stores the control program and various fixed data, as well as song data including lyric data and accompaniment data.
[0015] The CPU 201 is equipped with a timer 210 used in this embodiment, which counts the progress of an automatic performance in the electronic keyboard instrument 100, for example.
[0016] The sound source LSI 204 reads out musical tone waveform data from, for example, a waveform ROM (not shown) in accordance with sound generation control data 216 from the CPU 201, and outputs it as musical tone data 218 to the D / A converter 211. The sound source LSI 204 has the ability to generate a maximum of 256 voices simultaneously.
[0017] When the voice synthesis LSI 205 is provided by the CPU 201 with lyric text data (performance lyric data), data specifying each pitch corresponding to each lyric (performance pitch data), and data relating to the singing style (performance style data) as performance singing voice data 215, the voice synthesis LSI 205 synthesizes the corresponding singing voice data 217 and outputs it to the D / A converter 212.
[0018] A key scanner 206 constantly scans the key-on / key-off states of the keyboard 101 in FIG. 1 and the switch operation states of the first switch panel 102 and the second switch panel 103, and issues an interrupt to the CPU 201 to notify it of state changes.
[0019] The LCD controller 208 is an integrated circuit (IC) that controls the display state of the LCD 104 .
[0020] 3 is a block diagram showing an example of the configuration of a voice synthesis unit and a voice learning unit in this embodiment. Here, a voice synthesis unit 302 is built into the electronic keyboard instrument 100 as one function executed by the voice synthesis LSI 205 in FIG.
[0021] The voice synthesis unit 302 synthesizes and outputs singing voice data 217 by inputting performance-time singing voice data 215 including information on lyrics, pitch, and singing style instructed by the CPU 201 via the key scanner 206 in Fig. 2 based on key depression on the keyboard 101 in Fig. 1 by automatic lyrics playback (hereinafter referred to as "song playback") processing described later. At this time, the processor of the voice synthesis unit 302 inputs the performance-time singing voice data 215 including lyrics information generated by the CPU 201 in response to operation of any of the multiple keys (operators) on the keyboard 101, pitch information associated with any of the keys, and information on singing style to the performance-time singing voice analysis unit 307, inputs the performance-time language feature sequence 316 output from the performance-time singing voice analysis unit 307 to the trained acoustic model stored in the acoustic model unit 306, and performs vocalization processing to output singing voice data 217 inferring the singer's singing voice based on the spectrum information 318 and sound source information 319 output by the acoustic model unit 306.
[0022] For example, as shown in Fig. 3, the voice learning unit 301 may be implemented as a function executed by a server computer 300 that exists externally and is separate from the electronic keyboard instrument 100 of Fig. 1. Alternatively, although not shown in Fig. 3, the voice learning unit 301 may be built into the electronic keyboard instrument 100 as a function executed by the voice synthesis LSI 205 of Fig. 2 if the latter has sufficient processing power.
[0023] The speech training unit 301 and the speech synthesis unit 302 in FIG. 2 are implemented based on the technology of “statistical speech synthesis based on deep learning” described in Non-Patent Document 1 below, for example.
[0024] (Non-Patent Document 1) Kei Hashimoto, Shinji Takagi, "Statistical speech synthesis based on deep learning," Journal of the Acoustical Society of Japan, Vol. 73, No. 1 (2017), pp. 55-62.
[0025] As shown in FIG. 3, for example, a voice training unit 301 in FIG. 2, which is a function executed by an external server computer 300, includes a training singing voice analysis unit 303, a training acoustic feature extraction unit 304, and a model training unit 305.
[0026] In the voice learning unit 301, for example, a recording of a voice of a singer singing a plurality of songs of an appropriate genre is used as the learning singing voice data 312. Also, as the learning singing voice data 311, text data of lyrics of each song (learning lyrics data), data specifying each pitch corresponding to each lyric (learning pitch data), and data showing how to sing the learning singing voice data 312 (learning performance style data) are prepared. As the learning performance style data, the time intervals during which the learning pitch data are sequentially specified are measured, and each data indicating the sequentially measured time intervals is specified.
[0027] The training singing voice analysis unit 303 receives training singing voice data 311 including training lyric data, training pitch data, and training performance style data, and analyzes the data. As a result, the training singing voice analysis unit 303 estimates and outputs a training language feature sequence 313 which is a discrete numerical sequence expressing phonemes, pitches, and singing style corresponding to the training singing voice data 311.
[0028] The training acoustic feature extraction unit 304 inputs and analyzes training singing voice data 312 collected via a microphone or the like by a singer singing lyrics corresponding to the training singing voice data 311 in sync with the input of the training singing voice data 311. As a result, the training acoustic feature extraction unit 304 extracts a training acoustic feature sequence 314 representing features of voice corresponding to the training singing voice data 312, and outputs it as teacher data.
[0029] The model learning unit 305 calculates the learning language feature sequence 313 (which is TIFF0007578156000001.tif10168) and the acoustic model (this TIFF0007578156000002.tif11142) to obtain the training acoustic feature sequence 314 (which is The probability that TIFF0007578156000003.tif12147 is generated (let's put this as An acoustic model that maximizes TIFF0007578156000005.tif10111 is estimated by machine learning. That is, the relationship between the linguistic feature sequence, which is text, and the acoustic feature sequence, which is speech, is represented by a statistical model called an acoustic model.
number
[0030] Where: TIFF0007578156000007.tif11146 shows an operation to calculate the argument, written below it, that gives the maximum value for the function written to the right of it.
[0031] The model learning unit 305 learns an acoustic model calculated as a result of machine learning using the calculation shown in formula (1). The learning result data 315 representing TIFF0007578156000008.tif10111 is output.
[0032] For example, as shown in Fig. 3, the learning result data 315 may be stored in the ROM 202 of the control system of the electronic keyboard instrument 100 of Fig. 2 when the electronic keyboard instrument 100 of Fig. 1 is shipped from the factory, and may be loaded from the ROM 202 of Fig. 2 to an acoustic model section 306 (to be described later) in the voice synthesis LSI 205 when the electronic keyboard instrument 100 is powered on. Alternatively, the learning result data 315 may be downloaded to the acoustic model section 306 (to be described later) in the voice synthesis LSI 205 via a network interface 219 from the Internet or a USB (Universal Serial Bus) cable (not shown) by a performer operating a second switch panel 103 of the electronic keyboard instrument 100, as shown in Fig. 3. Alternatively, the trained acoustic model may be implemented as hardware using a field-programmable gate array (FPGA) or the like, separately from the voice synthesis LSI 205, and used as the acoustic model section.
[0033] The voice synthesis unit 302, which is a function executed by the voice synthesis LSI 205, includes a performance-time singing voice analysis unit 307, an acoustic model unit 306, and a vocalization model unit 308. The voice synthesis unit 302 executes a statistical voice synthesis process, which sequentially synthesizes and outputs singing voice data 217 corresponding to performance-time singing voice data 215 sequentially input during performance by predicting the singing voice data 217 using a statistical model called an acoustic model set in the acoustic model unit 306.
[0034] The performance-time singing voice analysis unit 307 inputs performance-time singing voice data 215 including information on performance-time lyric data (phonemes of lyrics corresponding to the lyric text), performance-time pitch data, and performance-time performance style data (singing style data) designated by the CPU 201 in Fig. 2 as the result of the performer's performance in sync with the automatic performance, and analyzes the data. As a result, the performance-time singing voice analysis unit 307 analyzes and outputs a performance-time linguistic feature sequence 316 expressing phonemes, parts of speech, words, pitches, and singing style corresponding to the performance-time singing voice data 215.
[0035] The acoustic model unit 306 receives a performance-time linguistic feature sequence 316, and estimates and outputs a performance-time acoustic feature sequence 317, which is an acoustic model parameter corresponding to the performance-time linguistic feature sequence 316. That is, the acoustic model unit 306 estimates and outputs the performance-time acoustic feature sequence 317, which is an acoustic model parameter corresponding to the performance-time linguistic feature sequence 316 received from the performance-time singing voice analysis unit 307, according to the following formula (2). TIFF0007578156000009.tif10149) and the acoustic model set as the learning result data 315 by machine learning in the model learning unit 305. Based on TIFF0007578156000010.tif10111, the performance acoustic feature sequence 317 (which is The probability of generating TIFF0007578156000011.tif12134 (let's put this as TIFF0007578156000012.tif13123) Estimate TIFF0007578156000013.tif10149.
number
[0036] The vocal model unit 308 inputs the performance-time acoustic feature sequence 317, and synthesizes and outputs the singing voice data 217 corresponding to the performance-time singing voice data 215 designated by the CPU 201. This singing voice data 217 is output from the D / A converter 212 in Fig. 2 via the mixer 213 and the amplifier 214, and is emitted from a speaker (not shown).
[0037] The acoustic features represented by the training acoustic feature sequence 314 and the performance acoustic feature sequence 317 include spectral information modeled on a human vocal tract and sound source information modeled on a human vocal cord. For example, Mel-cepstrum or Line Spectral Pairs (LSP) can be used as the spectral information (parameters). For example, a fundamental frequency (F0) and a power value indicating the pitch frequency of human voice can be used as the sound source information. The vocalization model unit 308 includes a sound source generation unit 309 and a synthesis filter unit 310. The sound source generation unit 309 is a part that models the human vocal cords, and by sequentially inputting a sequence of sound source information 319 input from the acoustic model unit 306, generates sound source signal data consisting of, for example, pulse train data (in the case of voiced phonemes) periodically repeated at the fundamental frequency (F0) and power value included in the sound source information 319, or white noise data (in the case of unvoiced phonemes) having the power value included in the sound source information 319, or a mixture of these data. The synthesis filter unit 310 is a part that models the human vocal tract, and forms a digital filter that models the vocal tract based on a series of spectrum information 318 sequentially input from the acoustic model unit 306, and generates and outputs singing voice data 321, which is digital signal data, using sound source signal data input from the sound source generation unit 309 as excitation source signal data.
[0038] The sampling frequency for the training singing voice data 312 and the singing voice data 217 is, for example, 16 KHz (kilohertz). When mel-cepstral parameters obtained by mel-cepstral analysis are adopted as the spectral parameters included in the training acoustic feature series 314 and the performance acoustic feature series 317, the update frame period is, for example, 5 msec (milliseconds). Furthermore, in the case of mel-cepstral analysis, the analysis window length is 25 msec, the window function is a Blackman window, and the analysis order is 24th.
[0039] 3, a method of using a hidden Markov model (HMM) or a method of using a deep neural network (DNN) can be adopted as an acoustic model represented by the learning result data 315 set in the acoustic model unit 306. Specific embodiments of these are disclosed in the above-mentioned Patent Document 1, and therefore detailed description thereof will be omitted in this application.
[0040] By using statistical voice synthesis processing consisting of a voice learning unit 301 and a voice synthesis unit 302 as illustrated in FIG. 3, the acoustic model unit 306 is equipped with a trained acoustic model that has learned the singing voice of a certain singer, and by sequentially inputting singing voice data 215 during performance, which includes the lyrics to be played back and the pitches specified by the performer to be pressed, an electronic keyboard instrument 100 is realized that outputs singing voice data 217 of a certain singer singing well.
[0041] Here, in singing voices, there is usually a difference in the way of singing between a melody of a fast passage and a melody of a slow passage. FIG. 4 is an explanatory diagram showing an example of notation on which the singing is based. FIG. 4(a) shows an example of a musical score of a lyric melody of a fast passage, and FIG. 4(b) shows an example of a musical score of a lyric melody of a slow passage. In these examples, the pattern of pitch change is similar, but FIG. 4(a) shows a continuous notation of sixteenth notes (note length is a quarter of a quarter note), while FIG. 4(b) shows a continuous notation of quarter notes. Therefore, the speed of changing the pitch is four times faster in the notation of FIG. 4(a) than in FIG. 4(b). In a song with a fast passage, the consonant part of the singing voice must be shortened in order to sing (perform) well. Conversely, in a song with a slow passage, the singing (perform) can be performed with higher expressiveness by making the consonant part of the singing voice longer. As mentioned above, even if the pitch change pattern is the same, differences in the length of each note in the singing melody (quarter notes, eighth notes, sixteenth notes, etc.) will result in differences in singing (performance) speed, but it goes without saying that even if the exact same musical score is sung (performed), there will be differences in performance speed if the tempo is changed during performance. In the following explanation, the time interval between notes (pronunciation speed) caused by the above two factors will be referred to as "performance tempo" to distinguish it from the tempo of a normal piece of music.
[0042] FIG. 5 is a diagram showing a waveform change of a singing voice caused by a difference in the performance tempo as illustrated in FIG. 4. The example shown in FIG. 5 shows an example of a waveform of a singing voice when the sound of / ga / is pronounced. The sound of / ga / is a sound that combines the consonant / g / and the vowel / a / . The duration (time length) of the consonant part is usually about several tens of milliseconds to about 200 milliseconds. Here, FIG. 5(a) shows an example of a singing voice waveform when a fast passage is sung, and FIG. 5(b) shows an example of a singing voice waveform when a slow passage is sung. The difference between the waveforms of FIG. 5(a) and (b) is that the length of the consonant / g / part is different. When a fast passage is sung, the pronunciation time length of the consonant part is short as shown in FIG. 5(a), and conversely, when a slow passage is sung, the pronunciation time length of the consonant part is long as shown in FIG. 5(b). When singing fast passages, consonants are not sung clearly and priority is given to the speed at which pronunciation begins, but in slower passages, consonants are often enunciated longer and more clearly, which increases the clarity of the words.
[0043] In order to reflect the difference in performance tempo in the change of singing voice data as described above, in the statistical voice synthesis process including the voice training unit 301 and the voice synthesis unit 302 in this embodiment illustrated in FIG. 3, training lyric data indicating lyrics and training pitch data indicating pitch are added to the training singing voice data 311 inputted in the voice training unit 301, and the training performance style data indicating the singing style is included in the training performance style data. The training singing voice analysis unit 303 in the voice training unit 301 analyzes such training singing voice data 311 to generate a training language feature sequence 313. Then, the model training unit 305 in the voice training unit 301 performs machine learning using the training language feature sequence 313. As a result, the model training unit 305 outputs a trained acoustic model including information on the performance tempo as training result data 315, and the trained acoustic model can be stored in the acoustic model unit 306 in the voice synthesis unit 302 of the voice synthesis LSI 205. As the training performance style data, the time intervals at which the training pitch data are sequentially designated are measured, and each performance tempo data indicating the sequentially measured time intervals is designated. In this way, the model training unit 305 in this embodiment can perform training to derive a trained acoustic model that takes into account differences in performance tempo due to singing styles.
[0044] Meanwhile, in the voice synthesis unit 302 including the acoustic model unit 306 in which the trained acoustic model described above is set, the performance-time lyric data indicating the lyrics and the performance-time pitch data indicating the pitch are added to the performance-time singing data 215, and the performance-time performance form data indicating the singing style can be included in the performance-time performance form data. The performance-time singing voice analysis unit 307 in the voice synthesis unit 302 generates a performance-time language feature sequence 316 by analyzing the performance-time singing voice data 215. Then, the acoustic model unit 306 in the voice synthesis unit 302 inputs the performance-time language feature sequence 316 into the trained acoustic model, thereby outputting the corresponding spectrum information 318 and sound source information 319, which are supplied to the synthesis filter unit 310 and sound source generation unit 309 in the voicing model unit 308, respectively. As a result, the vocalization model unit 308 can output the singing voice data 217 reflecting the change in the length of consonants due to the difference in the performance tempo depending on the singing style as shown in Fig. 5 (a) and (b). That is, it becomes possible to infer the appropriate singing voice data 217 that matches the change in the performance speed between notes that changes in real time.
[0045] FIG. 6 is a block diagram showing an example of the configuration of a lyric output unit, a pitch designation unit, and a performance style output unit which are realized by the CPU 201 in FIG. 2 as the control processing functions illustrated in the flowcharts of FIG. 8 to FIG. 11 to be described later, in order to generate the above-mentioned performance-time singing voice data 215.
[0046] The lyric output unit 601 outputs each performance-time lyric data 609 indicating the lyrics at the time of performance by including it in each performance-time singing voice data 215 output to the voice synthesis LSI 205 in Fig. 2. Specifically, the lyric output unit 601 sequentially reads each timing data 605 from the song playback music data 604 previously loaded from the ROM 202 to the RAM 203 by the CPU 201 in Fig. 2, while sequentially reading each lyric data (lyric text) 608 from each event data 606 stored as music data 604 in combination with each timing data 605 according to the timing indicated by each timing data 605, and sets each of them as performance-time lyric data 609.
[0047] The pitch designation unit 602 outputs each performance pitch data 610, which indicates each pitch designated in accordance with the output of each lyric during performance, by including it in each performance singing voice data 215 output to the voice synthesis LSI 205 of Fig. 2. Specifically, while sequentially reading each timing data 605 in the song data 604 for song playback loaded in the RAM 203, if the performer presses any key on the keyboard 101 of Fig. 1 at the timing indicated by each timing data 605 and pitch information of the pressed key is input via the key scanner 206, the pitch designation unit 602 sets the pitch information as performance pitch data 610. Furthermore, when the performer does not press any key on the keyboard 101 of FIG. 1 at the timing indicated by each timing data 605, the pitch designation unit 602 sets pitch data 607 in the event data 606 stored as song data 604 in combination with that timing data 605 as playing pitch data 610.
[0048] The performance style output unit 603 outputs performance style data 611 indicating the singing style, which is the performance style during performance, by including it in each performance singing voice data 215 to be output to the voice synthesis LSI 205 in FIG.
[0049] Specifically, when the performer sets the performance tempo mode to free mode on the first switch panel 102 of FIG. 1 as described later, the performance form output unit 603 sequentially measures the time intervals at which pitches are specified by the performer's key depressions during performance, and sets each performance tempo data indicating the sequentially measured time intervals as each performance-time performance form data 611.
[0050] On the other hand, when the performer has not set the performance tempo mode to the free mode on the first switch panel 102 of FIG. 1 as described later, the performance form output section 603 sets each performance tempo data corresponding to each time interval indicated by each timing data 605 sequentially read out from the song data 604 for reproducing the song loaded in the RAM 203 as each performance form data 611.
[0051] In addition, when the performer performs a performance tempo adjust setting on the first switch panel 102 of FIG. 1 to intentionally change the performance tempo mode as described later, the performance form output unit 603 intentionally changes the values of each performance tempo data obtained sequentially as described above based on the value of the performance tempo adjust setting, and sets each changed performance tempo data as performance form data 611 during performance.
[0052] In this manner, the functions of the lyric output unit 601, pitch designation unit 602 and performance style output unit 603 executed by the CPU 201 in FIG. 2 can generate performance-time singing voice data 215 including performance-time lyric data 609, performance-time pitch data 610 and performance-time performance style data 611 at the timing when a key press event occurs due to a performer's key press operation or song playback, and issue this to the voice synthesis unit 302 in the voice synthesis LSI 205 having the configuration in FIG. 2 or FIG. 3.
[0053] The operation of the embodiment of the electronic keyboard instrument 100 of Fig. 1 and Fig. 2 using the statistical voice synthesis process described in Fig. 3 to Fig. 6 will be described in detail below. Fig. 7 is a diagram showing a detailed data structure example of the music data read from the ROM 202 of Fig. 2 to the RAM 203 in this embodiment. This data structure example complies with the format of a standard MIDI file, which is one of the file formats for MIDI (Musical Instrument Digital Interface). This music data is made up of data blocks called chunks. Specifically, the music data is made up of a header chunk at the beginning of the file, followed by a track chunk 1 in which lyric data for the lyric part is stored, and a track chunk 2 in which performance data for the accompaniment part is stored.
[0054] The header chunk consists of four values: ChunkID, ChunkSize, FormatType, NumberOfTrack, and TimeDivision. ChunkID is a 4-byte ASCII code "4D 54 68 64" (numbers are hexadecimal) corresponding to the four half-width characters "MThd" which indicates that it is a header chunk. ChunkSize is 4-byte data indicating the data length of the FormatType, NumberOfTrack, and TimeDivision parts excluding ChunkID and ChunkSize in the header chunk, and the data length is fixed to 6 bytes: "00 00 00 06" (numbers are hexadecimal). In this embodiment, FormatType is 2-byte data "00 01" (numbers are hexadecimal) which means Format 1 which uses multiple tracks. In this embodiment, NumberOfTrack is 2-byte data "00 02" (numbers are hexadecimal) which indicates that two tracks corresponding to the lyrics part and the accompaniment part are used. The TimeDivision is data indicating a time base value that indicates the resolution per quarter note, and in this embodiment, is 2-byte data "01 E0" that indicates 480 in decimal (numbers are in hexadecimal).
[0055] Track chunk 1 indicates the lyric part and corresponds to the music data 604 in Fig. 6, and is made up of a performance data set (0≦i≦L-1) consisting of a ChunkID, a ChunkSize, and DeltaTime_1[i] corresponding to the timing data 605 in Fig. 6, and Event_1[i] corresponding to the event data 606 in Fig. 6. Track chunk 2 corresponds to the accompaniment part and is made up of a performance data set (0≦j≦M-1) consisting of a ChunkID, a ChunkSize, and DeltaTime_2[i] which is the timing data of the accompaniment part, and Event_2[j] which is the event data of the accompaniment part.
[0056] Each ChunkID in track chunks 1 and 2 is a 4-byte ASCII code "4D 54 72 6B" (numbers are hexadecimal) that corresponds to the four half-width characters "MTrk" that indicate that it is a track chunk. Each ChunkSize in track chunks 1 and 2 is 4-byte data that indicates the data length of the part of each track chunk excluding the ChunkID and ChunkSize.
[0057] DeltaTime_1[i], which is timing data 605 in Fig. 6, is variable-length data of 1 to 4 bytes indicating the waiting time (relative time) from the execution time of Event_1[i-1], which is the immediately preceding event data 606 in Fig. 6. Similarly, DeltaTime_2[i], which is timing data of the accompaniment part, is variable-length data of 1 to 4 bytes indicating the waiting time (relative time) from the execution time of Event_2[i-1], which is the immediately preceding event data of the accompaniment part.
[0058] 6, Event_1[i] is a meta event having two pieces of information, the vocal text of the lyrics and the pitch, in track chunk 1 / lyrics part of this embodiment. Event_2[i] is event data for the accompaniment part, and is a MIDI event instructing note-on or note-off of the accompaniment sound, or a meta event instructing the beat of the accompaniment sound, in track chunk 2 / accompaniment part.
[0059] In each performance data set DeltaTime_1[i] and Event_1[i] of track chunk 1 / lyric part, the event data 606, Event_1[i], is executed after waiting the timing data 605, DeltaTime_1[i], from the execution time of the immediately preceding event data 606, Event_1[i-1], thereby realizing the progression of song playback. Meanwhile, in each performance data set DeltaTime_2[i] and Event_2[i] of track chunk 2 / accompaniment part, the event data Event_2[i] is executed after waiting the timing data, DeltaTime_2[i], from the execution time of the immediately preceding event data, Event_2[i-1], thereby realizing the progression of automatic accompaniment.
[0060] 8 is a main flow chart showing an example of control processing of the electronic musical instrument in this embodiment. This control processing is, for example, an operation in which the CPU 201 in FIG.
[0061] The CPU 201 first executes an initialization process (step S801), and then repeatedly executes a series of processes from steps S802 to S808.
[0062] In this repeat process, the CPU 201 first executes a switch process (step S802). Here, the CPU 201 executes a process corresponding to a switch operation of the first switch panel 102 or the second switch panel 103 in Fig. 1 based on an interrupt from the key scanner 206 in Fig. 2. The details of the switch process will be described later with reference to the flowchart in Fig. 10.
[0063] Next, the CPU 201 executes keyboard processing to determine whether any key on the keyboard 101 in Fig. 1 has been operated based on an interrupt from the key scanner 206 in Fig. 2 and to process the determination (step S803). In the keyboard processing, the CPU 201 outputs musical tone control data 216 instructing the sound source LSI 204 in Fig. 2 to start or stop sound generation in response to the performer's operation of pressing or releasing any key. In the keyboard processing, the CPU 201 also executes processing to calculate the time interval from the previous key pressed to the current key pressed as performance tempo data. Details of the keyboard processing will be described later with reference to the flowchart in Fig. 11.
[0064] Next, the CPU 201 processes data to be displayed on the LCD 104 of Fig. 1, and executes display processing to display the data on the LCD 104 via the LCD controller 208 of Fig. 2 (step S804). Data to be displayed on the LCD 104 includes, for example, lyrics corresponding to the singing voice data 217 to be performed, melody and accompaniment scores corresponding to the lyrics, and various setting information.
[0065] Next, the CPU 201 executes a song playback process (step S805). In the song playback process, the CPU 201 generates performance-time singing voice data 215 including lyrics, vocal pitch, and performance tempo for operating the voice synthesis LSI 205 based on song playback, and issues the data to the voice synthesis LSI 205. The song playback process will be described in detail later with reference to the flowchart of FIG.
[0066] Next, the CPU 201 executes a sound source process (step S806). In the sound source process, the CPU 201 executes control processes such as envelope control of musical tones being generated in the sound source LSI 204.
[0067] Next, the CPU 201 executes a voice synthesis process (step S807). In the voice synthesis process, the CPU 201 controls the execution of voice synthesis by the voice synthesis LSI 205.
[0068] Finally, the CPU 201 judges whether or not the performer has turned off the power by pressing a power-off switch (not shown) (step S808). If the judgment in step S808 is NO, the CPU 201 returns to the process in step S802. If the judgment in step S808 is YES, the CPU 201 ends the control process shown in the flowchart of FIG. 8 and turns off the power to the electronic keyboard instrument 100.
[0069] 9(a), (b), and (c) are flowcharts respectively showing detailed examples of the initialization processing in step S801 in FIG. 8, the tempo change processing in step S1002 in FIG. 10 (to be described later) in the switch processing in step S802 in FIG. 8, and the song start processing in step S1006 in FIG. 10.
[0070] First, in FIG. 9(a) showing a detailed example of the initialization process of step S801 in FIG. 8, the CPU 201 executes the initialization process of TickTime. In this embodiment, the progression of lyrics and automatic accompaniment progress in units of time called TickTime. The time base value specified as the TimeDivision value in the header chunk of the music data exemplified in FIG. 7 indicates the resolution of a quarter note, and if this value is 480, for example, a quarter note has a time length of 480 TickTime. The values of the waiting times DeltaTime_1[i] and DeltaTime_2[i] in each track chunk of the music data exemplified in FIG. 7 are also counted in units of TickTime. Here, how many seconds one TickTime actually occupies depends on the tempo specified for the music data. Now, assuming that the tempo value is Tempo [beats / minute] and the above time base value is TimeDivision, the number of seconds of TickTime is calculated by the following formula (3).
[0071] TickTime[seconds]=60 / Tempo / TimeDivision (3)
[0072] In the initialization process shown in the flowchart of Fig. 9(a), the CPU 201 first calculates TickTime [seconds] by performing an arithmetic operation corresponding to the above formula (10) (step S901). It is assumed that the tempo value Tempo is initially stored in the ROM 202 of Fig. 2 as a predetermined value, for example, 60 [beats / second]. Alternatively, the tempo value at the time of the previous termination may be stored in the non-volatile memory.
[0073] Next, the CPU 201 sets a timer interrupt for the TickTime [seconds] calculated in step S901 for the timer 210 in Fig. 2 (step S902). As a result, an interrupt for song playback and automatic accompaniment (hereinafter referred to as an "auto-playing interrupt") is generated for the CPU 201 every time the TickTime [seconds] elapses in the timer 210. Therefore, in the auto-playing interrupt process (Fig. 12, described later) executed by the CPU 201 based on this auto-playing interrupt, a control process is executed to progress the song playback and automatic accompaniment for each tickTime.
[0074] Next, the CPU 201 executes other initialization processes such as the initialization of the RAM 203 in Fig. 2 (step S903) After that, the CPU 201 ends the initialization process in step S801 in Fig. 8 illustrated in the flowchart in Fig. 9(a).
[0075] 9B and 9C will be described later. Fig. 10 is a flowchart showing a detailed example of the switch process in step S802 in Fig. 8.
[0076] First, the CPU 201 judges whether or not the tempo of the lyric progression and automatic accompaniment has been changed by the tempo change switch in the first switch panel 102 in Fig. 1 (step S1001). If the judgment is YES, the CPU 201 executes a tempo change process (step S1002). The details of this process will be described later with reference to Fig. 9(b). If the judgment in step S1001 is NO, the CPU 201 skips the process in step S1002.
[0077] Next, the CPU 201 determines whether or not any song has been selected on the second switch panel 103 of FIG. 1 (step S1003). If the determination is YES, the CPU 201 executes a song reading process (step S1004). This process is a process for reading song data having the data structure described in FIG. 7 from the ROM 202 of FIG. 2 into the RAM 203. Note that the song reading process may be performed not only during performance but also before the start of performance. After this, data access to track chunk 1 or 2 in the data structure exemplified in FIG. 7 is performed on the song data read into the RAM 203. If the determination in step S1003 is NO, the CPU 201 skips the process in step S1004.
[0078] Next, the CPU 201 determines whether or not the song start switch has been operated on the first switch panel 102 of Fig. 1 (step S1005). If the determination is YES, the CPU 201 executes song start processing (step S1006). Details of this processing will be described later with reference to Fig. 9(c). If the determination in step S1005 is NO, the CPU 201 skips the processing in step S1006.
[0079] Next, the CPU 201 judges whether or not the free mode switch has been operated on the first switch panel 102 in FIG. 1 (step S1007). If the judgement is YES, the CPU 201 executes a free mode set process to change the value of the variable FreeMode on the RAM 203 (step S1008). The free mode switch, for example, operates as a toggle, and the value of the variable FreeMode is initially set to, for example, a value of 1 in step S903 in FIG. 9. When the free mode switch is pressed in this state, the value of the variable FreeMode becomes 0, and when pressed again, the value becomes 1. In this manner, the value of the variable FreeMode alternates between 0 and 1 each time the free mode switch is pressed. When the value of the variable FreeMode is 1, the free mode is set, and when the value is 0, the free mode setting is released. If the judgement is NO in step S1007, the CPU 201 skips the process in step S1008.
[0080] Next, the CPU 201 determines whether or not the performance tempo adjust switch has been operated on the first switch panel 102 of Fig. 1 (step S1009). If the determination is YES, the CPU 201 executes a performance tempo adjust setting process to change the value of the variable ShiinAdjust on the RAM 203 to a value designated by the numeric keys on the first switch panel 102 following the operation of the performance tempo adjust switch (step S1010). The value of the variable ShiinAdjust is initially set to a value of 0, for example, in step S903 of Fig. 9. If the determination is NO in step S1009, the CPU 201 skips the process of step S1010.
[0081] Finally, CPU 201 determines whether any other switch has been operated on first switch panel 102 or second switch panel 103 in Fig. 1, and executes a process corresponding to each switch operation (step S1011). After that, CPU 201 ends the switch process of step S802 in Fig. 8, which is exemplified in the flowchart of Fig. 10.
[0082] Fig. 9(b) is a flowchart showing a detailed example of the tempo change process in step S1002 in Fig. 10. As described above, when the tempo value is changed, the TickTime [seconds] is also changed. In the flowchart in Fig. 9(b), the CPU 201 executes control processing related to the change of this TickTime [seconds].
[0083] First, the CPU 201 calculates TickTime [seconds] (step S911) by the calculation process corresponding to the above-mentioned formula (3), in the same manner as in step S901 in Fig. 9(a) executed in the initialization process of step S801 in Fig. 8. Note that it is assumed that the tempo value Tempo is a value changed by the tempo change switch in the first switch panel 102 in Fig. 1 and stored in the RAM 203 or the like.
[0084] Next, the CPU 201 sets a timer interrupt for the timer 210 in Fig. 2 based on the TickTime [seconds] calculated in step S911 (step S912), in the same manner as in step S902 in Fig. 9(a) executed in the initialization process in step S801 in Fig. 8. Thereafter, the CPU 201 ends the tempo change process in step S1002 in Fig. 10, exemplified in the flowchart in Fig. 9(b).
[0085] FIG. 9C is a flow chart showing a detailed example of the song start process in step S1006 of FIG.
[0086] First, the CPU 201 initializes the values of timing data variables DeltaT_1 (track chunk 1) and DeltaT_2 (track chunk 2) on the RAM 203, which are used to count the relative time from the occurrence time of the immediately preceding event in units of TickTime during the progress of automatic performance, to 0. Next, the CPU 201 initializes the values of a variable AutoIndex_1 on the RAM 203, which is used to specify the value of i of each of the performance data sets DeltaTime_1[i] and Event_1[i] (1≦i≦L-1) in the track chunk 1 of the music data illustrated in Fig. 7, and a variable AutoIndex_2 on the RAM 203, which is used to specify the value of j of each of the performance data sets DeltaTime_2[j] and Event_2[j] (1≦j≦M-1) in the track chunk 2, to 0 (step S921). In the example of FIG. 7, therefore, the first performance data sets DeltaTime_1[0] and Event_1[0] in track chunk 1, and the first performance data sets DeltaTime_2[0] and Event_2[0] in track chunk 2 are referenced as the initial state.
[0087] Next, the CPU 201 initializes the value of the variable SongIndex in the RAM 203, which indicates the current song position, to a Null value (step S922). Although the Null value is usually defined as 0, since the index number may be 0, in this embodiment the Null value is defined as -1.
[0088] Furthermore, the CPU 201 initializes the value of a variable SongStart in the RAM 203, which indicates whether the lyrics and accompaniment are to progress (=1) or not (=0), to 1 (to progress) (step S923).
[0089] After that, the CPU 201 determines whether or not the performer has set up the playback of an accompaniment in accordance with the playback of the lyrics through the first switch panel 102 in FIG. 1 (step S924).
[0090] If the determination in step S924 is YES, the CPU 201 sets the value of the variable Bansou in the RAM 203 to 1 (with accompaniment) (step S925). Conversely, if the determination in step S924 is NO, the CPU 201 sets the value of the variable Bansou to 0 (without accompaniment) (step S926). After the process of step S925 or S926, the CPU 201 ends the song start process of step S1006 in Fig. 10, exemplified in the flowchart of Fig. 9(c).
[0091] Fig. 11 is a flowchart showing a detailed example of the keyboard process in step S803 in Fig. 8. First, the CPU 201 determines whether or not any key on the keyboard 101 in Fig. 1 has been operated via the key scanner 206 in Fig. 2 (step S1101).
[0092] If the determination in step S1101 is NO, the CPU 201 ends the keyboard process in step S803 in FIG. 8, which is illustrated in the flowchart of FIG.
[0093] If the determination in step S1101 is YES, the CPU 201 determines whether a key has been pressed or released (step S1102).
[0094] If it is determined in step S1102 that a key has been released, the CPU 201 instructs the voice synthesis LSI 205 to mute the vocalization of the singing voice data 217 corresponding to the released pitch (or key number) (step S1113). In response to this instruction, the voice synthesis unit 302 in the voice synthesis LSI 205 (Fig. 3) stops vocalization of the corresponding singing voice data 217. Thereafter, the CPU 201 ends the keyboard process in step S803 in Fig. 8, as exemplified in the flowchart in Fig. 11.
[0095] If it is determined in step S1102 that a key has been pressed, the CPU 201 determines the value of the variable FreeMode in the RAM 203 (step S1103). The value of this variable FreeMode is set in step S1008 of Fig. 10 described above, and when the variable FreeMode has a value of 1, the free mode is set, and when the variable FreeMode has a value of 0, the free mode is released.
[0096] If it is determined in step 1103 that the value of the variable free mode is 0 and the free mode setting is cancelled, then the CPU 201 sets a value calculated by the calculation process exemplified by the following equation (4) using each timing data 605, DeltaTime_1[AutoIndex_1] described later, which is sequentially read out from the song data 604 for song playback loaded into the RAM 203, as described above in the explanation of the performance form output unit 603 in FIG. 6, to the variable PlayTempo on the RAM 203 indicating the performance tempo corresponding to the performance form data 611 during performance in FIG. 6 (step S1109).
[0097] PlayTempo=(1 / DeltaTime_1[AutoIndex_1]) ×Prescribed coefficient (4)
[0098] In the present embodiment, the predetermined coefficient in formula (4) is the TimeDivision value of the song data × 60. That is, if the TimeDivision value is 480, when DeltaTime_1[AutoIndex_1] is 480, PlayTempo is 60 (corresponding to a normal tempo of 60). When DeltaTime_1[AutoIndex_1] is 240, PlayTempo is 120 (corresponding to a normal tempo of 120).
[0099] When the free mode is disabled, the performance tempo is set in synchronization with the timing information of the song playback.
[0100] If it is determined in step S1103 that the value of the variable FreeMode is 1, the CPU 201 further determines whether the value of the variable NoteOnTime in the RAM 203 is a Null value (step S1104). When song playback starts, for example in step S903 in Fig. 9, the value of the variable NoteOnTime is initially set to a Null value, and after song playback starts, the current time of the timer 210 in Fig. 2 is sequentially set in step S1110, which will be described later.
[0101] If the determination in step S1104 is YES at the start of song playback, the performance tempo cannot be determined from the performer's key depressions, so CPU 201 sets the value calculated by the arithmetic processing illustrated in equation (4) above using DeltaTime_1[AutoIndex_1], which is timing data 605 in RAM 203, to variable PlayTempo in RAM 203 (step S1109). In this way, at the start of song playback, the performance tempo is provisionally set in synchronization with the timing information for song playback.
[0102] If the song playback has started and the determination in step S1104 is NO, the CPU 201 first subtracts the value of the variable NoteOnTime in RAM 203 indicating the time of the previous key press from the current time indicated by the timer 210 in FIG. 2, and sets the difference time to the variable DeltaTime in RAM 203 (step S1105).
[0103] Next, the CPU 201 determines whether the value of the variable DeltaTime indicating the difference in time from the previous key press to the current key press is smaller than a predetermined maximum time that is regarded as simultaneous key presses due to a chord performance (step S1106).
[0104] If the determination in step S1106 is YES, that is, if it is determined that the current key presses are simultaneous key presses resulting from a chord performance (chord), the CPU 201 does not execute a process for determining the performance tempo, but proceeds to the process of step S1110, which will be described later.
[0105] If the determination in step S1106 is NO, that is, the current key presses are not simultaneous key presses resulting from a chord performance, the CPU 201 further determines whether the value of the variable DeltaTime, which indicates the difference in time from the previous key press to the current key press, is greater than the minimum time for which the performance is deemed to have been interrupted (step S1107).
[0106] If the determination in step S1107 is YES, that is, the key is pressed after a brief pause in performance (the beginning of a performance phrase), the performance tempo of the performance phrase cannot be determined, so the CPU 201 sets the value calculated by the arithmetic processing exemplified by the above-mentioned equation (4) using DeltaTime_1[AutoIndex_1], which is timing data 605 in RAM 203, to the variable PlayTempo in RAM 203 (step S1109). In this way, when the key is pressed after a brief pause in performance (the beginning of a performance phrase), the performance tempo is provisionally set in synchronization with the timing information of the song playback.
[0107] If the determination in step S1107 is NO, that is, if it is determined that the current key press is neither a simultaneous key press due to a chord performance (chord) nor a key press at the beginning of a performance phrase, the CPU 201 sets a value obtained by multiplying the reciprocal of the variable DeltaTime, which indicates the difference in time from the previous key press to the current key press, by a predetermined coefficient, as exemplified in the following equation (5), to the variable PlayTempo in the RAM 203, which indicates the performance tempo corresponding to the performance pattern data 611 during performance in FIG. 6 (step S1108).
[0108] PlayTempo = (1 / DeltaTime) × given coefficient (5)
[0109] When the variable DeltaTime indicating the time difference between the previous key pressing and the current key pressing is small, the value of the performance tempo PlayTempo becomes large (the performance tempo becomes faster), the performance phrase is considered to be a fast passage, and the voice synthesis unit 302 in the voice synthesis LSI 205 infers the voice waveform of the singing voice data 217 with a short consonant time length as shown in Fig. 5(a) . On the other hand, when the variable DeltaTime indicating the time difference is large, the value of the performance tempo becomes small (the performance tempo becomes slower), the performance phrase is considered to be a slow passage, and the voice synthesis unit 302 infers the voice waveform of the singing voice data 217 with a long consonant time length as shown in Fig. 5(b) .
[0110] After the processing of step S1108 described above, after the processing of step S1109 described above, or after the determination of step S1106 described above becomes YES, CPU 201 sets variable NoteOnTime in RAM 203, which indicates the time of the previous key press, to the current time indicated by timer 210 in FIG. 2 (step S1110).
[0111] Finally, the CPU 201 adds the value of the variable PlayTempo in the RAM 203 indicating the performance tempo determined in step S1108 or S1109 to the value of the variable ShiinAdjust in the RAM 203 in which the performance tempo adjust value intentionally set by the performer (see step S1010 in FIG. 10), and sets the resultant value as the new value of the variable PlayTempo (step S1111). After that, the CPU 201 ends the keyboard process in step S803 in FIG. 8 illustrated in the flowchart in FIG. 11.
[0112] By the process of step S1111, the performer can intentionally adjust the time length of the consonant part in the singing voice data 217 synthesized by the voice synthesis unit 302. The performer may want to adjust the singing style depending on the piece of music or his / her preference. For example, when a performer wants to play a piece of music with short sounds overall and crisply, he / she may want the consonants to be shortened and the voice to be pronounced as if sung quickly, and conversely, when a performer wants to play a piece of music overall slowly, he / she may want the voice to be pronounced in such a way that the breath of the consonants can be clearly heard as if sung slowly. Therefore, in this embodiment, the performer can change the value of the variable ShiinAdjust by operating the performance tempo adjust switch on the first switch panel 102 of FIG. 1, for example, and adjust the value of the variable PlayTempo based on this, thereby synthesizing the singing voice data 217 that reflects the performer's intention. In addition to the switch operation, the value of ShiinAdjust can also be finely controlled at any timing during the piece of music by operating a pedal using a variable resistor connected to the electronic keyboard instrument 100 with the foot.
[0113] The performance tempo value set in the variable PlayTempo by the above keyboard processing is set as part of the performance vocal data 215 in the song playback processing described later (see step S1305 in FIG. 13 described later) and issued to the voice synthesis LSI 205.
[0114] In the above keyboard processing, particularly the processing from step S1103 to step S1109 and step S1111 correspond to the function of the performance style output unit 603 in FIG.
[0115] Fig. 12 is a flow chart showing a detailed example of an automatic performance interrupt process executed based on an interrupt (see step S902 in Fig. 9(a) or step S912 in Fig. 9(b)) generated every TickTime [seconds] in the timer 210 in Fig. 2. The following process is executed for the performance data sets of track chunks 1 and 2 of the song data exemplified in Fig. 7.
[0116] First, the CPU 201 executes a series of processes (steps S1201 to S1206) corresponding to track chunk 1. First, the CPU 201 determines whether or not the SongStart value is 1 (see step S1006 in FIG. 10 and step S923 in FIG. 9), that is, whether or not the progression of lyrics and accompaniment is instructed (step S1201).
[0117] If it is determined that the progression of lyrics and accompaniment has not been instructed (NO in step S1201), the CPU 201 does not progress the lyrics and accompaniment, and simply ends the automatic performance interrupt process illustrated in the flowchart of FIG. 12.
[0118] If it is determined that progression of lyrics and accompaniment has been instructed (the determination in step S1201 is YES), the CPU 201 determines whether the value of the variable DeltaT_1 on the RAM 203, which indicates the relative time from the occurrence time of the previous event related to track chunk 1, matches DeltaTime_1[AutoIndex_1] on the RAM 203, which is timing data 605 (FIG. 6) indicating the waiting time of the performance data set to be executed, which is indicated by the value of the variable AutoIndex_1 on the RAM 203 (step S1202).
[0119] If the determination in step S1202 is NO, the CPU 201 increments the value of the variable DeltaT_1, which indicates the relative time from the occurrence time of the previous event, by +1 for track chuck 1, and advances the time by one TickTime unit corresponding to the current interrupt (step S1203).Then, the CPU 201 proceeds to step S1207, which will be described later.
[0120] If the determination in step S1202 is YES, the CPU 201 stores the value of the variable AutoIndex_1, which indicates the position of the next song event to be executed in track chunk 1, in the variable SongIndex on the RAM 203 (step S1204).
[0121] Furthermore, the CPU 201 increments the value of the variable AutoIndex_1, which is used to refer to the performance data set in the track chunk 1, by +1 (step S1205).
[0122] Furthermore, the CPU 201 resets the value of the variable DeltaT_1, which indicates the relative time from the occurrence time of the currently referenced song event for track chunk 1, to 0 (step S1206). After that, the CPU 201 proceeds to the process of step S1207.
[0123] Next, the CPU 201 executes a series of processes (steps S1207 to S1213) corresponding to the track chunk 2. First, the CPU 201 determines whether the value of the variable DeltaT_2 in the RAM 203, which indicates the relative time from the occurrence time of the previous event related to the track chunk 2, matches the timing data DeltaTime_2[AutoIndex_2] in the RAM 203 of the performance data set to be executed, which is indicated by the value of the variable AutoIndex_2 in the RAM 203 (step S1207).
[0124] If the determination in step S1207 is NO, the CPU 201 increments the value of the variable DeltaT_2, which indicates the relative time from the occurrence time of the previous event, by +1 for track check 2, and advances the time by one TickTime unit corresponding to the current interrupt (step S1208). Thereafter, the CPU 201 ends the automatic performance interrupt process illustrated in the flowchart of FIG.
[0125] If the determination in step S1207 is YES, the CPU 201 determines whether the value of the variable Bansou in the RAM 203, which instructs accompaniment playback, is 1 (with accompaniment) (without accompaniment) (step S1209) (see steps S924 to S926 in FIG. 9C).
[0126] If the determination in step S1209 is YES, the CPU 201 executes the process indicated by the event data Event_2[AutoIndex_2] in the RAM 203 related to the accompaniment of the track chuck 2 indicated by the value of the variable AutoIndex_2 (step S1210). If the process indicated by the event data Event_2[AutoIndex_2] executed here is, for example, a note-on event, an instruction to generate an accompaniment tone is issued to the sound source LSI 204 in Fig. 2 based on the key number and velocity specified by the note-on event. On the other hand, if the process indicated by the event data Event_2[AutoIndex_2] is, for example, a note-off event, an instruction to mute the currently generated accompaniment tone is issued to the sound source LSI 204 in Fig. 2 based on the key number specified by the note-off event.
[0127] On the other hand, if the determination in step S1209 is NO, CPU 201 skips step S1210 and does not execute the processing indicated by the event data Event_2[AutoIndex_2] related to the current accompaniment, but proceeds to the next step S1211 to execute only the control processing for advancing the event progression in synchronization with the lyrics.
[0128] After step S1210 or if the determination in step S1209 is NO, the CPU 201 increments the value of the variable AutoIndex_2, which is used to refer to the performance data set for the accompaniment data on the track chunk 2, by +1 (step S1211).
[0129] Next, the CPU 201 resets the value of a variable DeltaT_2, which indicates the relative time from the occurrence time of the event data currently executed for track chunk 2, to 0 (step S1212).
[0130] Then, the CPU 201 determines whether the value of the timing data DeltaTime_2[AutoIndex_2] in the RAM 203 of the performance data set on track chunk 2 to be executed next, which is indicated by the value of the variable AutoIndex_2, is 0, i.e., whether the event is to be executed simultaneously with the current event (step S1213).
[0131] If the determination in step S1213 is NO, the CPU 201 ends the current automatic performance interrupt process exemplified in the flowchart of FIG.
[0132] If the determination in step S1213 is YES, the CPU 201 returns to the process of step S1209 and repeats the control process for the event data Event_2[AutoIndex_2] in the RAM 203 of the performance data set to be executed next on the track chunk 2 indicated by the value of the variable AutoIndex_2. The CPU 201 repeats the processes from steps S1209 to S1213 the same number of times as the number of times that they will be executed simultaneously this time. The above process sequence is executed when multiple note-on events are sounded at the same time, such as in a chord.
[0133] FIG. 13 is a flow chart showing a detailed example of the song playback process in step S805 of FIG.
[0134] First, in step S1204 of the automatic performance interrupt process of Fig. 12, the CPU 201 determines whether a new value other than Null has been set to the variable SongIndex in the RAM 203 and whether the song is being played back (step S1301). At the start of a song, the variable SongIndex is initially set to a Null value in step S922 of Fig. 9(c) described above, and whenever the timing for playing back vocals arrives, the determination in step S1202 of the automatic performance interrupt process of Fig. 12 becomes YES, and in the following step S1204, a valid value is set to the variable AutoIndex_1 indicating the position of the next song event to be executed in track chunk 1, and further, each time the song playback process illustrated in the flowchart of Fig. 13 is executed once, the variable SongIndex is reset to a Null value again in step S1307 described later. In other words, whether a valid value other than Null has been set to the variable SongIndex indicates whether the current timing is the timing for playing back the song.
[0135] If the determination in step S1301 becomes YES, that is, if it is now time to play back the song, the CPU 201 determines whether or not a new key pressed by the performer on the keyboard 101 in FIG. 1 has been detected by the keyboard process in step S803 in FIG. 8 (step S1302).
[0136] If the determination in step S1302 is YES, the CPU 201 sets the pitch designated by the performer's key depression as the vocal pitch in a register (not shown) or a variable in the RAM 203 (step S1303).
[0137] On the other hand, if step S1301 determines that the current time has come to play the song and the determination in step S1302 is NO, i.e., that a new key press has not been detected at this time, the CPU 201 reads pitch data (corresponding to pitch data 607 in event data 606 in FIG. 6) from song event data Event_1[SongIndex] in track chunk 1 of the song data on RAM 203 indicated by the variable SongIndex on RAM 203, and sets this pitch data as the vocal pitch in a register (not shown) or a variable on RAM 203 (step S1304).
[0138] Next, the CPU 201 reads out a lyric string (corresponding to the lyric data 608 in the event data 606 in FIG. 6) from the song event Event_1[SongIndex] in the track chunk 1 of the music data in the RAM 203 indicated by the variable SongIndex in the RAM 203. The CPU 201 then sets the read out lyric string (corresponding to the performance-time lyric data 609 in FIG. 6), the vocal pitch (corresponding to the performance-time pitch data 610 in FIG. 6) acquired in step S1303 or S1304, and the performance tempo (corresponding to the performance-time performance form data 611 in FIG. 6) acquired in step S1111 in FIG. 10 corresponding to step S803 in FIG. 8 described above) in the performance-time singing voice data 215 in which the performance tempo acquired in the variable PlayTempo in the RAM 203 in step S1111 in FIG. 10 corresponding to step S803 in FIG. 8 described above is set in a register (not shown) or a variable in the RAM 203 (step S1305).
[0139] Next, the CPU 201 issues the playing-time singing voice data 215 created in step S1305 to the voice synthesis unit 302 in Fig. 3 of the voice synthesis LSI 205 in Fig. 2 (step S1306). As described with reference to Figs. 3 to 6, the voice synthesis LSI 205 infers, synthesizes and outputs singing voice data 217 that appropriately sings the lyrics specified by the playing-time singing voice data 215 in real time in accordance with the key pressed on the keyboard 101 by the performer specified by the playing-time singing voice data 215 or the pitch automatically specified as the pitch data 607 (see Fig. 6) by song playback, and further in accordance with the performance tempo (singing style) specified by the playing-time singing voice data 215.
[0140] Finally, the CPU 201 clears the value of the variable SongIndex to a null value, so that subsequent timings are not song playback timings (step S1307). After that, the CPU 201 ends the song playback process of step S805 in Fig. 8, as exemplified in the flowchart in Fig. 13.
[0141] In the above song playback process, the processes in steps S1302 to S1304 in particular correspond to the function of the pitch designation unit 602 in Fig. 6. Moreover, the process in step S1305 in particular corresponds to the function of the lyrics output unit 601 in Fig. 6.
[0142] With the embodiment described above, depending on the type of song being played and the phrase being played, the length of time that the consonants in the vocal voice are pronounced can be made long and expressive and vivid for a slow passage with few notes, and can be made short and crisp for a fast tempo or a performance with many notes, making it possible to obtain a change in timbre that matches the phrase being played.
[0143] The embodiment described above is an embodiment of an electronic musical instrument that generates singing voice data, but as another embodiment, an embodiment of an electronic musical instrument that generates wind instrument sounds or string instrument sounds can also be implemented. In this case, the acoustic model unit corresponding to the acoustic model unit 306 in FIG. 3 is machine-trained with training pitch data that specifies a pitch, teacher data corresponding to the training acoustic data that indicates the sound of a certain sound source of a wind instrument or string instrument that corresponds to the pitch, and training performance style data that indicates the performance style (e.g., performance tempo) of the training acoustic data, and stores a trained acoustic model that outputs acoustic model parameters corresponding to the input pitch data and performance style data. In addition, the pitch designation unit (corresponding to the pitch designation unit 602 in FIG. 6) outputs performance-time pitch data that indicates the pitch designated by the performer's performance operation during performance. Furthermore, the performance style output unit (corresponding to the performance style output unit 603 in FIG. 6) outputs performance-time performance style data that indicates the performance style during the performance described above, for example, the performance tempo. Then, the pronunciation model unit (corresponding to the vocalization model unit 308 in FIG. 3) synthesizes and outputs musical tone data that infers the voice of a certain sound source based on the acoustic model parameters that are output by inputting the above-mentioned playing pitch data and playing style data into the trained acoustic model stored in the acoustic model unit during playing. In such an embodiment of the electronic musical instrument, for example, in a piece with a fast passage, pitch data that shortens the blow sound at the beginning of playing a wind instrument or the speed at which the bow hits the string of a string instrument when the bow is rubbed against the string is inferred and synthesized, thereby enabling a crisp performance. Conversely, in a piece with a slow passage, pitch data that lengthens the duration of the blow sound at the beginning of playing a wind instrument or the sound of the bow hitting the string when the bow is rubbed against the string is inferred and synthesized, thereby enabling a performance with high expressiveness.
[0144] In the embodiment described above, when it is not possible to estimate the speed of a performance phrase, such as the first key press or the first key press of a performance phrase, the strength with which the keys are pressed (the velocity value at the time the key is pressed) may be used as the basis for calculating the performance tempo value, taking advantage of the fact that consonants and the onset parts of sounds tend to be shorter when sung or played strongly, and longer when sung or played softly.
[0145] The voice synthesis method that can be adopted as the speech model unit 308 in FIG. 3 is not limited to the cepstrum voice synthesis method, and various voice synthesis methods including the LSP voice synthesis method can be adopted.
[0146] Furthermore, as a voice synthesis method, any voice synthesis method may be adopted that uses statistical voice synthesis processing based on machine learning, such as a statistical voice synthesis processing using an HMM acoustic model, a statistical voice synthesis processing using a DNN acoustic model, or an acoustic model that combines HMM and DNN.
[0147] In the embodiment described above, the performance-time lyric data 609 is provided as pre-stored song data 604, but text data obtained by performing voice recognition on the content sung by the performer in real time may also be provided in real time as lyric information.
[0148] The following supplementary notes are further disclosed regarding the above-described embodiment. (Appendix 1) a pitch designation unit that outputs performance-time pitch data designated during performance; a performance style output unit for outputting performance style data indicating the performance style at the time of playing; a sound generation model unit which synthesizes and outputs musical tone data corresponding to the playing pitch data and the playing performance style data based on acoustic model parameters inferred by inputting the playing pitch data and the playing performance style data to a trained acoustic model during the playing; An electronic musical instrument comprising: (Appendix 2) a lyrics output unit that outputs lyrics data at the time of performance, the lyrics data indicating the lyrics at the time of performance; a pitch designation unit that outputs performance-time pitch data that is designated in accordance with the output of the lyrics during the performance; a performance style output unit for outputting performance style data indicating the performance style at the time of playing; a vocalization model unit which synthesizes and outputs singing voice data corresponding to the lyrics data at the time of performance, the pitch data at the time of performance, and the performance style data at the time of performance based on an acoustic model parameter inferred by inputting the lyrics data at the time of performance, the pitch data at the time of performance, and the performance style data at the time of performance into a trained acoustic model; An electronic musical instrument comprising: (Appendix 3) 3. The electronic musical instrument according to claim 1, wherein the performance pattern output unit sequentially measures time intervals at which the pitch is specified during the performance, and sequentially outputs performance tempo data indicating the sequentially measured time intervals as the performance pattern data during the performance. (Appendix 4) 4. The electronic musical instrument according to claim 3, wherein the performance style output unit includes a change means for allowing a player to intentionally change the performance tempo data obtained in sequence. (Appendix 5) Electronic musical instrument processors, Outputs the pitch data specified during performance, outputting performance style data indicating the performance style at the time of the performance; synthesizing and outputting musical tone data corresponding to the performance pitch data and the performance style data during the performance based on acoustic model parameters inferred by inputting the performance pitch data and the performance style data during the performance into a trained acoustic model during the performance; A method for controlling an electronic musical instrument to execute processing. (Appendix 6) Electronic musical instrument processors, Outputting lyrics data indicating lyrics at the time of performance; outputting performance pitch data designated in accordance with the output of the lyrics during the performance; outputting the performance style data during the performance, which indicates the performance style during the performance; During the performance, the performance-time lyric data, the performance-time pitch data, and the performance-time performance form data are inputted into a trained acoustic model, and based on the acoustic model parameters inferred, singing voice data corresponding to the performance-time lyric data, the performance-time pitch data, and the performance-time performance form data are synthesized and output. A method for controlling an electronic musical instrument to execute processing. (Appendix 7) Electronic musical instrument processors, Outputs the pitch data specified during performance, outputting performance style data indicating the performance style at the time of the performance; synthesizing and outputting musical tone data corresponding to the performance pitch data and the performance style data during the performance based on acoustic model parameters inferred by inputting the performance pitch data and the performance style data during the performance into a trained acoustic model during the performance; A program for executing a process. (Appendix 8) Electronic musical instrument processors, Outputting lyrics data indicating lyrics at the time of performance; outputting performance pitch data designated in accordance with the output of the lyrics during the performance; outputting the performance style data during the performance, which indicates the performance style during the performance; During the performance, the performance-time lyric data, the performance-time pitch data, and the performance-time performance form data are inputted into a trained acoustic model, and based on the acoustic model parameters inferred, singing voice data corresponding to the performance-time lyric data, the performance-time pitch data, and the performance-time performance form data are synthesized and output. A program for executing a process. [Explanation of symbols]
[0149] 100 Electronic Keyboard Instruments 101 Keyboard 102 First Switch Panel 103 Second Switch Panel 104 LCD 200 Control System 201 CPU 202 ROM 203 RAM 204 Sound source LSI 205 Voice Synthesis LSI 206 Key Scanner 208 LCD Controller 209 System Bus 210 Timer 211, 212 D / A converter 213 Mixer 214 Amplifier 215 Vocal Data 216 Sound Control Data 217 Singing voice data 218 Musical Data 219 Network Interface 300 Server Computers 301 Audio Learning Department 302 Speech synthesis unit 303 Learning vocal analysis section 304 Acoustic Feature Extraction for Training 305 Model Learning Department 306 Acoustic Model Section 307 Singing voice analysis section during performance 308 Vocal Model Section 309 Sound source generation section 310 Synthesis filter section 311 Vocal Data for Learning 312 Singing voice data for training 313 Training linguistic feature sequence 314 Acoustic feature sequence for training 315 Learning outcome data 316 Performance linguistic information series 317 Acoustic feature sequence during performance 318 Spectral Information 319 Sound Source Information 601 Lyrics output section 602 Pitch specification section 603 Performance Format Output Unit 604 Song Data 605 Timing Data 606 Event Data 607 Pitch Data 608 Lyrics Data 609 Lyrics data during performance 610 Pitch data during performance 611 Performance form data
Claims
1. a pitch designation unit that outputs performance-time pitch data designated during performance; a performance style output unit for determining a performance tempo data value based on a velocity value when an initial pitch is designated by the performance during the performance, and outputting the performance tempo data as performance style data during the performance, which indicates a performance style during the performance; a sound generation model unit which synthesizes and outputs musical tone data corresponding to the playing pitch data and the playing performance style data based on acoustic model parameters inferred by inputting the playing pitch data and the playing performance style data to a trained acoustic model during the playing; An electronic musical instrument comprising:
2. a lyrics output unit that outputs lyrics data at the time of performance, the lyrics data indicating the lyrics at the time of performance; a pitch designation unit that outputs performance-time pitch data that is designated in accordance with the output of the lyrics during the performance; a performance style output unit for determining a performance tempo data value based on a velocity value when an initial pitch is designated by the performance during the performance, and outputting the performance tempo data as performance style data during the performance, which indicates a performance style during the performance; a vocalization model unit which synthesizes and outputs singing voice data corresponding to the lyrics data at the time of performance, the pitch data at the time of performance, and the performance style data at the time of performance based on an acoustic model parameter inferred by inputting the lyrics data at the time of performance, the pitch data at the time of performance, and the performance style data at the time of performance into a trained acoustic model; An electronic musical instrument comprising:
3. 3. The electronic musical instrument according to claim 1, wherein the performance tempo data is data showing a faster performance tempo as the velocity value when the initial pitch is designated is larger.
4. 3. The electronic musical instrument according to claim 2, wherein the duration of consonant parts sounded in said singing voice data is adjusted in accordance with said performance tempo data.
5. Electronic musical instrument processors, Outputs the pitch data specified during performance, determining a value of performance tempo data based on a velocity value when an initial pitch is designated by the performance during said performance, and outputting said performance tempo data as performance form data during said performance, which indicates a performance form during said performance; synthesizing and outputting musical tone data corresponding to the performance pitch data and the performance style data during the performance based on acoustic model parameters inferred by inputting the performance pitch data and the performance style data during the performance into a trained acoustic model during the performance; A method for controlling an electronic musical instrument to execute processing.
6. Electronic musical instrument processors, Outputting lyrics data indicating lyrics at the time of performance; outputting performance pitch data designated in accordance with the output of the lyrics during the performance; determining a value of performance tempo data based on a velocity value when an initial pitch is designated by the performance during said performance, and outputting said performance tempo data as performance form data during said performance, which indicates a performance form during said performance; During the performance, the performance-time lyric data, the performance-time pitch data, and the performance-time performance form data are inputted into a trained acoustic model, and based on the acoustic model parameters inferred, singing voice data corresponding to the performance-time lyric data, the performance-time pitch data, and the performance-time performance form data are synthesized and output. A method for controlling an electronic musical instrument to execute processing.
7. Electronic musical instrument processors, Outputs the pitch data specified during performance, determining a value of performance tempo data based on a velocity value when an initial pitch is designated by the performance during said performance, and outputting said performance tempo data as performance form data during said performance, which indicates a performance form during said performance; synthesizing and outputting musical tone data corresponding to the performance pitch data and the performance style data during the performance based on acoustic model parameters inferred by inputting the performance pitch data and the performance style data during the performance into a trained acoustic model during the performance; A program for executing a process.
8. Electronic musical instrument processors, Outputting lyrics data indicating lyrics at the time of performance; outputting performance pitch data designated in accordance with the output of the lyrics during the performance; determining a value of performance tempo data based on a velocity value when an initial pitch is designated by the performance during said performance, and outputting said performance tempo data as performance form data during said performance, which indicates a performance form during said performance; During the performance, the performance-time lyric data, the performance-time pitch data, and the performance-time performance form data are inputted into a trained acoustic model, and based on the acoustic model parameters inferred, singing voice data corresponding to the performance-time lyric data, the performance-time pitch data, and the performance-time performance form data are synthesized and output. A program for executing a process.
Citation Information
Patent Citations
JP1982009994U
Tempo controller
JP1985070495A
JP1990035199U
Musical performance data generation device and program for realizing musical performance data generation method
JP2015075574A
Singing voice synthesis device and singing voice synthesis method
JP2017107228A
