Electronic musical instrument, method and program

Abstract

To appropriately control the progression of phrases related to a musical performance.SOLUTION: An electronic musical instrument comprises: a plurality of performance keys respectively associated with tone pitch data different from each other; and a processor. The processor determines syllable positions included in a phrase on the basis of operations to the performance keys, included in a first range, of the plurality of performance keys and directs the pronunciation of syllables corresponding to the determined syllable positions on the basis of operations to the performance keys, included in a second range, of the plurality of performance keys.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present disclosure relates to electronic musical instruments, methods and programs. [Background technology]

[0002] In recent years, the use of synthetic voices has been expanding. In this context, if there were an electronic musical instrument that could not only play automatically, but also progress lyrics according to the key presses of the user (player) and output synthetic voice corresponding to the lyrics, it would be desirable because it would enable more flexible expression of synthetic voices.

[0003] For example, Patent Document 1 discloses a technique for controlling lyrics to be sounded in response to the performance of a keyboard instrument, using a controller separate from the keyboard instrument. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2018 / 123456 Summary of the Invention [Problem to be solved by the invention]

[0005] However, introducing a dedicated controller as in Patent Document 1 is problematic in terms of user operation, as it is difficult to easily enjoy the pronunciation of lyrics using synthetic voice.

[0006] Therefore, one of the objects of the present disclosure is to provide an electronic musical instrument, method, and program that can appropriately control the progression of phrases (for example, lyrics) that are being played. [Means for solving the problem]

[0007] In an electronic musical instrument according to an embodiment of the present disclosure, a plurality of syllables included in a phrase are included in a first range that is assigned to each syllable. All white keysThe musical instrument is provided with a plurality of performance operators including a plurality of first performance operators and a plurality of second performance operators included in a second range, each of which is associated with different pitch data, and a processor, wherein the processor determines a syllable position based on an operation on one of the first performance operators, and instructs the pronunciation of a syllable corresponding to the determined syllable position based on an operation on the second performance operator. Effect of the Invention

[0008] According to one aspect of the present disclosure, the phrase progression during performance can be appropriately controlled. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of the external appearance of an electronic musical instrument 10 according to an embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of a hardware configuration of a control system 200 of the electronic musical instrument 10 according to an embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an example of the configuration of the voice learning unit 301 according to an embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of the waveform data output unit 211 according to an embodiment. [Diagram 5] FIG. 5 is a diagram showing another example of the waveform data output unit 211 according to an embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of division of a keyboard into key ranges for controlling syllable positions according to an embodiment. [Figure 7] 7A-7C are diagrams showing examples of syllables assigned to the control key range. [Figure 8] FIG. 8 is a diagram illustrating an example of a flowchart of a lyrics progression control method according to an embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of a flowchart of a syllable position control process according to an embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of a flowchart of a performance control process according to an embodiment. [Figure 11] FIG. 11 is a diagram illustrating an example of a flowchart of a syllable progression determination process according to an embodiment. [Figure 12] FIG. 12 is a diagram illustrating an example of a flowchart of the syllable changing process according to an embodiment. [Figure 13] 13A and 13B are diagrams showing an example of the appearance of a key in the control key area. [Figure 14] FIG. 14 is a diagram showing an example of a tablet terminal that implements the lyrics progress control method according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the following description, the same parts are denoted by the same reference numerals. Since the same parts have the same names, functions, etc., detailed description will not be repeated.

[0011] (Electronic Musical Instruments) 1 is a diagram showing an example of the appearance of an electronic musical instrument 10 according to an embodiment. The electronic musical instrument 10 may include a switch (button) panel 140b, a keyboard 140k, a pedal 140p, a display 150d, and a speaker 150s.

[0012] The electronic musical instrument 10 is a device for receiving input from a user via controls such as a keyboard and switches, and for controlling musical performance, lyric progression, etc. The electronic musical instrument 10 is a device for controlling musical performance, lyric progression, etc. The device may be an electronic musical instrument (such as an electronic piano or synthesizer) or an analog musical instrument equipped with a sensor or the like and configured to have the functions of the above-mentioned controls.

[0013] The switch panel 140b may include switches for operating the volume specification, settings such as sound source and tone, song (accompaniment) selection, song playback start / stop, song playback settings (tempo, etc.), and the like.

[0014] The keyboard 140k may have multiple keys as performance operators. The pedal 140p may be a sustain pedal that sustains the sound of a pressed key while the pedal is pressed, or a pedal for operating an effector that processes the tone, volume, etc.

[0015] In the present disclosure, the terms "sustain pedal," "pedal," "foot switch," "controller," "switch," "button," "touch panel," etc. may be interchangeable. In the present disclosure, "pressing a pedal" may be interchangeable with "operating a controller."

[0016] The keys may be called performance operators, pitch operators, timbre operators, direct operators, first operators, etc. The pedals may be called non-performance operators, non-pitch operators, non-timbre operators, indirect operators, second operators, etc.

[0017] The display 150d may display lyrics, musical scores, various setting information, etc. The speaker 150s may be used to emit sounds generated by playing.

[0018] The electronic musical instrument 10 may be capable of generating and converting at least one of MIDI messages (events) and Open Sound Control (OSC) messages.

[0019] Electronic musical instrument 10 may be referred to as controller 10, syllable progression controller 10, or the like.

[0020] The electronic musical instrument 10 may communicate with a network (such as the Internet) via at least one of wired and wireless communication (e.g., Long Term Evolution (LTE), 5th generation mobile communication system New Radio (5G NR), Wi-Fi (registered trademark), etc.).

[0021] The electronic musical instrument 10 may hold in advance vocal data (which may be called lyric text data, lyric information, etc.) relating to lyrics for which progress is to be controlled, or may transmit and / or receive the vocal data via a network. The vocal data may be text written in a musical score description language (e.g., MusicXML), may be written in a MIDI data storage format (e.g., Standard MIDI File (SMF) format), or may be text provided in a normal text file. The vocal data may be vocal data 215, which will be described later. In the present disclosure, the terms vocal, voice, sound, etc. may be interchangeable.

[0022] In addition, the electronic musical instrument 10 may obtain the content sung by the user in real time via a microphone or the like provided in the electronic musical instrument 10, and apply voice recognition processing to the content to obtain text data as singing voice data.

[0023] FIG. 2 is a diagram showing an example of a hardware configuration of a control system 200 of the electronic musical instrument 10 according to an embodiment.

[0024] The computer includes a central processing unit (CPU) 201, a read-only memory (ROM) 202, a random access memory (RAM) 203, a waveform data output unit 211, a key scanner 206 to which the switch (button) panel 140b, the keyboard 140k, and the pedal 140p of FIG. 1 are connected, and an LCD (Liquid Crystal Display) as an example of the display 150d of FIG. An LCD controller 208 to which an LCD (crystal display) is connected is connected to the system bus 209.

[0025] A timer 210 (which may be called a counter) for controlling performance may be connected to the CPU 201. The timer 210 may be used, for example, to count the progress of an automatic performance in the electronic musical instrument 10. The CPU 201 may be called a processor, and may include an interface with peripheral circuits, a control circuit, an arithmetic circuit, a register, and the like.

[0026] 1 by executing a control program stored in ROM 202 while using RAM 203 as a working memory. In addition to the control program and various fixed data, ROM 202 may also store vocal data, accompaniment data, and song data including these.

[0027] The waveform data output unit 211 may include a sound source LSI (large scale integrated circuit) 204, a voice synthesis LSI 205, etc. The sound source LSI 204 and the voice synthesis LSI 205 may be integrated into one LSI. A specific block diagram of the waveform data output unit 211 will be described later with reference to FIG. 3. Note that part of the processing of the waveform data output unit 211 may be performed by the CPU 201, or may be performed by a CPU included in the waveform data output unit 211.

[0028] The singing voice waveform data 217 and song waveform data 218 output from the waveform data output unit 211 are converted into an analog singing voice output signal and an analog musical sound output signal by D / A converters 212 and 213, respectively. The analog musical sound output signal and analog singing voice output signal may be mixed by a mixer 214, and the mixed signal may be amplified by an amplifier 215 and then output from the speaker 150s or an output terminal. The singing voice waveform data may be called singing voice synthesis data. Although not shown, the singing voice waveform data 217 and song waveform data 218 may be digitally synthesized, and then converted to analog by a D / A converter to obtain a mixed signal.

[0029] A key scanner (scanner) 206 constantly scans the key-on / key-off state of the keyboard 140k in FIG. 1, the switch operation state of the switch panel 140b, the pedal operation state of the pedal 140p, and the like, and issues an interrupt to the CPU 201 to notify it of state changes.

[0030] The LCD controller 208 is an IC (integrated circuit) that controls the display state of an LCD, which is an example of the display 150d.

[0031] It should be noted that the above system configuration is merely an example and is not limited to this. For example, the number of each circuit included is not limited to this. Electronic musical instrument 10 may have a configuration that does not include some circuits (mechanisms), or may have a configuration in which the function of one circuit is realized by multiple circuits. It may also have a configuration in which the functions of multiple circuits are realized by one circuit.

[0032] Furthermore, electronic musical instrument 10 may be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), etc., and some or all of the functional blocks may be realized by the hardware. For example, CPU 201 may be implemented by at least one of these pieces of hardware.

[0033] <Generating Acoustic Models> Fig. 3 is a diagram showing an example of the configuration of a voice training unit 301 according to an embodiment. The voice training unit 301 may be implemented as a function executed by a server computer 300 that is externally present and separate from the electronic musical instrument 10 of Fig. 1. The voice training unit 301 may be built into the electronic musical instrument 10 as a function executed by the CPU 201, the voice synthesis LSI 205, or the like.

[0034] The voice training unit 301 and the waveform data output unit 211 that realize the voice synthesis in the present disclosure may each be implemented based on, for example, a statistical voice synthesis technique based on deep learning.

[0035] The speech training unit 301 may include a training text analysis unit 303 , a training acoustic feature extraction unit 304 , and a model training unit 305 .

[0036] In the voice learning unit 301, for example, recordings of a singer singing a number of songs in an appropriate genre are used as the learning singing voice data 312. Also, as the learning singing voice data 311, lyrics texts of each song are prepared.

[0037] The training text analysis unit 303 receives training singing data 311 including lyrics text and analyzes the data. As a result, the training text analysis unit 303 estimates and outputs a training language feature sequence 313 which is a discrete numerical sequence expressing phonemes, pitches, etc. corresponding to the training singing data 311.

[0038] The training acoustic feature extraction unit 304 inputs and analyzes training singing voice data 312 collected via a microphone or the like by a singer singing lyrics text corresponding to the training singing voice data 311 in accordance with the input of the training singing voice data 311. As a result, the training acoustic feature extraction unit 304 extracts and outputs a training acoustic feature sequence 314 expressing voice features corresponding to the training singing voice data 312.

[0039] In the present disclosure, the training acoustic feature sequence 314 and the acoustic feature sequence corresponding to the acoustic feature sequence 317 described later include acoustic feature data (which may be called formant information, spectral information, etc.) that models the human vocal tract, and vocal cord sound source data (which may be called sound source information) that models the human vocal cords. For example, Mel-cepstrum, Line Spectral Pairs (LSP), etc. can be used as the spectral information. For example, a fundamental frequency (F0) and a power value that indicate the pitch frequency of human voice can be used as the sound source information.

[0040] The model learning unit 305 estimates, by machine learning, an acoustic model that maximizes the probability that the training acoustic feature sequence 314 is generated from the training language feature sequence 313. That is, the relationship between the language feature sequence, which is text, and the acoustic feature sequence, which is speech, is represented by a statistical model called an acoustic model. The model learning unit 305 outputs model parameters that represent the acoustic model calculated as a result of the machine learning as the learning result 315. Therefore, the acoustic model corresponds to a trained model.

[0041] As the acoustic model represented by the learning result 315 (model parameters), a hidden Markov model (HMM) may be used.

[0042] The HMM acoustic model may learn how the characteristic parameters of the vocal voice, such as the vibration of the vocal cords and the vocal tract characteristics, change over time when a singer sings lyrics according to a certain melody. More specifically, the HMM acoustic model may be a model of the spectrum, fundamental frequency, and their time structures calculated from the training vocal data, on a phoneme-by-phoneme basis.

[0043] First, the processing of the speech training unit 301 in Fig. 3 in which the HMM acoustic model is adopted will be described. A model training unit 305 in the speech training unit 301 may input the training language feature sequence 313 output by the training text analysis unit 303 and the training acoustic feature sequence 314 output by the training acoustic feature extraction unit 304, to train an HMM acoustic model that maximizes the likelihood.

[0044] The spectral parameters of singing voices can be modeled using a continuous HMM. On the other hand, the logarithmic fundamental frequency (F0) is a variable-dimensional time-series signal that takes continuous values ​​in voiced sections and has no value in unvoiced sections, so it cannot be directly modeled using a normal continuous or discrete HMM. Therefore, we use MSD-HMM (Multi-Space probability Distribution HMM), an HMM based on a probability distribution in a multi-space corresponding to variable dimensions, and simultaneously model the mel-cepstrum as a multidimensional Gaussian distribution for the spectral parameters, and the logarithmic fundamental frequency (F0) as a Gaussian distribution in one-dimensional space for voiced sounds and zero-dimensional space for unvoiced sounds.

[0045] It is also known that the characteristics of the phonemes that make up a singing voice vary under the influence of various factors, even if the acoustic characteristics of the phonemes are the same. For example, the spectrum and logarithmic fundamental frequency (F0) of a phoneme, which is a basic phonetic unit, differ depending on the singing style, tempo, or the lyrics and pitch of the preceding and following songs. Such factors that affect acoustic features are called contexts.

[0046] In the statistical speech synthesis process of an embodiment, an HMM acoustic model that takes context into consideration (context-dependent model) may be adopted to accurately model the acoustic features of speech. Specifically, the training text analysis unit 303 may output a training language feature sequence 313 that takes into consideration not only the phonemes and pitch of each frame, but also the immediately preceding and following phonemes, the current position, the immediately preceding and following vibratos, accents, etc. Furthermore, context clustering based on a decision tree may be used to improve the efficiency of the combination of contexts.

[0047] For example, the model training unit 305 may generate, as the training result 315, a state duration decision tree for determining state duration from a training language feature sequence 313 corresponding to the context of a large number of phonemes related to state duration extracted by the training text analysis unit 303 from the training singing data 311.

[0048] In addition, the model training unit 305 may generate, as the training result 315, a Mel-Cepstral parameter decision tree for determining Mel-Cepstral parameters from the training acoustic feature sequence 314 corresponding to a large number of phonemes related to the Mel-Cepstral parameters extracted by the training acoustic feature extraction unit 304 from the training singing voice data 312.

[0049] Furthermore, the model learning unit 305 may generate, as the learning result 315, a logarithmic fundamental frequency decision tree for determining the logarithmic fundamental frequency (F0) from the training acoustic feature sequence 314 corresponding to a large number of phonemes related to the logarithmic fundamental frequency (F0) extracted by the training acoustic feature extraction unit 304 from the training singing voice data 312. Note that the voiced and unvoiced sections of the logarithmic fundamental frequency (F0) may be modeled as one-dimensional and zero-dimensional Gaussian distributions, respectively, by the MSD-HMM corresponding to the variable dimension, and a logarithmic fundamental frequency decision tree may be generated.

[0050] Instead of or together with the HMM-based acoustic model, an acoustic model based on a deep neural network (DNN) may be adopted. In this case, the model learning unit 305 may generate model parameters representing a nonlinear conversion function of each neuron in the DNN from the language feature to the acoustic feature as the learning result 315. The DNN makes it possible to express the relationship between the language feature sequence and the acoustic feature sequence using a complex nonlinear conversion function that is difficult to express using a decision tree.

[0051] Furthermore, the acoustic models of the present disclosure are not limited to these, and any voice synthesis method may be adopted as long as it is a technology that uses statistical voice synthesis processing, such as an acoustic model that combines HMM and DNN.

[0052] The learning results 315 (model parameters) may be stored in the ROM 202 of the control system of the electronic musical instrument 10 of FIG. 2 when the electronic musical instrument 10 of FIG. 1 is shipped from the factory, as shown in FIG. 3, and may be loaded from the ROM 202 of FIG. 2 to a singing voice control unit 307 (described later) in the waveform data output unit 211 when the electronic musical instrument 10 of FIG. 1 is powered on.

[0053] The learning results 315 may be downloaded to the singing voice control unit 307 in the waveform data output unit 211 from an external source such as the Internet via the network interface 219 by the performer operating the switch panel 140b of the electronic musical instrument 10, as shown in FIG. 3, for example.

[0054] <Speech synthesis based on acoustic models> FIG. 4 is a diagram illustrating an example of the waveform data output unit 211 according to an embodiment.

[0055] The waveform data output unit 211 includes a processing unit (which may be called a text processing unit, a pre-processing unit, etc.) 306, a singing voice control unit (which may be called an acoustic model unit) 307, a sound source 308, a singing voice synthesis unit (which may be called a vocalization model unit) 309, etc.

[0056] The waveform data output unit 211 receives singing voice data 215 including lyrics and pitch information and lyrics control data instructed by the CPU 201 via the key scanner 206 in Fig. 2 based on key depressions of the keyboard 140k (performance operators) in Fig. 1, and synthesizes and outputs singing voice waveform data 217 corresponding to the lyrics and pitch. In other words, the waveform data output unit 211 executes a statistical voice synthesis process that synthesizes singing voice waveform data 217 corresponding to the singing voice data 215 including lyrics text by predicting using a statistical model called an acoustic model set in the singing voice control unit 307.

[0057] Furthermore, when playing back song data, the waveform data output unit 211 outputs song waveform data 218 corresponding to the corresponding song playback position. Here, the song data may correspond to accompaniment data (e.g., data on pitch, tone color, sound timing, etc., for one or more sounds), accompaniment and melody data, or may be called backtrack data, etc.

[0058] 2 as a result of a performer's performance (operation), the processing unit 306 inputs singing voice data 215 including information on the phonemes, pitch, etc. of the lyrics specified by the CPU 201 in Fig. 2, for example, and analyzes the data. The singing voice data 215 may include at least one of data (e.g., pitch data, note length data) of the nth note (which may also be called the nth note, the nth timing, etc.), data of the nth lyric (or syllable) corresponding to the nth note, data of the nth syllable, etc.

[0059] For example, the processing unit 306 may determine the presence or absence of lyric progression based on note on / off data, pedal on / off data, etc. acquired from the operation of the keyboard 140k and the pedal 140p, in accordance with a lyric progression control method described later, and acquire the singing voice data 215 corresponding to the syllables (lyrics) to be output. The processing unit 306 may then analyze the linguistic feature series 316 expressing phonemes, parts of speech, words, etc., corresponding to the pitch data designated by the key depression or the pitch data of the acquired singing voice data 215 and the character data of the acquired singing voice data 215, and output the linguistic feature series 316 to the singing voice control unit 307.

[0060] The singing voice data 215 may be information including at least one of the following: lyrics (characters), syllable type (start syllable, middle syllable, end syllable, etc.), corresponding pitch (correct pitch), and lyrics (character string) for each syllable. The singing voice data 215 may include information on singing voice data of the nth syllable corresponding to the nth (n=1, 2, 3, 4, ...) syllable.

[0061] The singing voice data 215 may include information (such as data in a specific audio file format, MIDI data, etc.) for playing an accompaniment (song data) corresponding to the lyrics. When the singing voice data is represented in SMF format, the singing voice data 215 may include a track chunk in which data related to the singing voice is stored, and a track chunk in which data related to the accompaniment is stored. The singing voice data 215 may be read from the ROM 202 to the RAM 203. The singing voice data 215 is stored in a memory (for example, the ROM 202, the RAM 203) before it is played.

[0062] The lyrics control data may be used to set vocal reproduction information corresponding to a syllable, as described later with reference to Fig. 12. The waveform data output unit 211 can control the timing of pronunciation based on the vocal reproduction information. For example, the processing unit 306 may adjust the language feature sequence 316 to be output to the vocal control unit 307 based on the syllable start frame indicated by the vocal reproduction information (for example, frames prior to the syllable start frame may not be output).

[0063] The singing control unit 307 estimates an acoustic feature sequence 317 corresponding to the language feature sequence 316 input from the processing unit 306 and the acoustic model set as the learning result 315, and outputs formant information 318 corresponding to the estimated acoustic feature sequence 317 to the singing synthesis unit 309.

[0064] For example, when an HMM acoustic model is adopted, the singing control unit 307 connects HMMs by referring to a decision tree for each context obtained by the language feature sequence 316, and predicts the acoustic feature sequence 317 (formant information 318 and vocal cord sound source data 319) with the maximum output probability from each of the connected HMMs.

[0065] When the DNN acoustic model is adopted, the singing control unit 307 may output the acoustic feature sequence 317 in units of frames for the phoneme sequence of the language feature sequence 316 input in units of frames. Note that the frame in the present disclosure may be, for example, 5 ms, 10 ms, etc.

[0066] In FIG. 4, the processing unit 306 obtains musical instrument sound data (pitch information) corresponding to the pitch of the pressed key from a memory (which may be the ROM 202 or the RAM 203 ), and outputs it to the sound source 308 .

[0067] The sound source 308 generates a sound source signal (which may be called instrument sound waveform data) of instrument sound data (pitch information) corresponding to the sound to be generated (note-on) based on the note-on / off data input from the processing unit 306, and outputs it to the singing voice synthesis unit 309. The sound source 308 may execute control processes such as envelope control of the sound to be generated.

[0068] The singing voice synthesis unit 309 forms a digital filter that models the vocal tract based on a series of formant information 318 sequentially input from the singing voice control unit 307. The singing voice synthesis unit 309 also uses the sound source signal input from the sound source 308 as an excitation source signal, applies the digital filter, and generates and outputs the singing voice waveform data 217 of a digital signal. In this case, the singing voice synthesis unit 309 may be called a synthesis filter unit.

[0069] The singing voice synthesis unit 309 may be capable of employing various voice synthesis methods, including the cepstrum voice synthesis method and the LSP voice synthesis method.

[0070] In the example of FIG. 4, the singing voice waveform data 217 to be output is a little less faithful than the singer's singing voice because the sound of a musical instrument is used as the sound source signal, but the singing voice retains both the atmosphere of the musical instrument sound and the vocal quality of the singer, so that effective singing voice waveform data 217 can be output.

[0071] The sound source 308 may operate to process the instrument sound waveform data and output the output of other channels as song waveform data 218. This makes it possible to produce accompaniment sounds using normal instrument sounds, or to produce instrument sounds for a melody line while simultaneously vocalizing the melody line.

[0072] 5 is a diagram showing another example of the waveform data output unit 211 according to an embodiment. Contents that overlap with those in FIG. 4 will not be described repeatedly.

[0073] 5 estimates an acoustic feature sequence 317 based on the acoustic model as described above. Then, the singing control unit 307 outputs formant information 318 corresponding to the estimated acoustic feature sequence 317 and vocal cord sound source data (pitch information) 319 corresponding to the estimated acoustic feature sequence 317 to the singing synthesis unit 309. The singing control unit 307 may estimate an estimate of the acoustic feature sequence 317 that maximizes the probability that the acoustic feature sequence 317 is generated.

[0074] The singing synthesis unit 309 may generate data (which may be called, for example, singing waveform data of the nth lyric corresponding to the nth note) for generating a signal by applying a digital filter that models the vocal tract based on the series of formant information 318 to, for example, a pulse train (in the case of voiced phonemes) that is periodically repeated at the fundamental frequency (F0) and power value included in the vocal cord sound source data 319 input from the singing control unit 307, or white noise (in the case of unvoiced phonemes) having the power value included in the vocal cord sound source data 319, or a signal obtained by mixing these, and outputting the data to the sound source 308.

[0075] The sound source 308 generates and outputs singing voice waveform data 217 of a digital signal from the singing voice waveform data of the nth lyric corresponding to the note to be sounded (note-on) based on the note-on / off data input from the processing unit 306.

[0076] In the example of Figure 5, the output vocal waveform data 217 is a signal that is completely modeled by the vocal control unit 307, as it is a sound source signal generated by the sound source 308 based on the vocal cord sound source data 319, and therefore it is possible to output vocal waveform data 217 that is very faithful to the singer's singing voice and has a natural singing voice.

[0077] In this way, the voice synthesis disclosed herein differs from existing vocoders (a technique in which spoken words are input through a microphone and replaced with instrument sounds for synthesis), in that it is possible to output synthetic voice by operating the keyboard without the user (performer) actually singing (in other words, without the user inputting voice signals to be pronounced in real time into electronic musical instrument 10).

[0078] As described above, by adopting the statistical voice synthesis processing technology as the voice synthesis method, it is possible to realize a memory capacity that is significantly smaller than that of the conventional voice unit synthesis method. For example, in an electronic musical instrument using the voice unit synthesis method, a memory with a storage capacity of several hundred megabytes is required for voice unit data, but in this embodiment, a memory with a storage capacity of only a few megabytes is sufficient to store the model parameters of the learning result 315. This makes it possible to realize a lower-priced electronic musical instrument, and makes it possible for a wider range of users to use a high-quality singing voice performance system.

[0079] Furthermore, in the conventional segment data method, since the segment data needs to be adjusted manually, it takes a huge amount of time (years) and effort to create data for singing performance, but in the creation of model parameters of the learning result 315 for the HMM acoustic model or DNN acoustic model according to this embodiment, almost no data adjustment is required, so the creation time and effort is reduced to a fraction of that. This also makes it possible to realize a lower-priced electronic musical instrument.

[0080] In addition, general users can use the learning function built into the server computer 300 and voice synthesis LSI 205 available as a cloud service to learn their own voice, the voice of a family member, or the voice of a celebrity, and use that as a model voice to play a singing voice on an electronic musical instrument. In this case, it is also possible to realize a singing voice performance that is much more natural and high quality than before, on a lower-cost electronic musical instrument.

[0081] (Lyric progression control method) A lyric progression control method according to an embodiment of the present disclosure will be described below. Note that the lyric progression control in the present disclosure may be read as performance control, performance, and the like.

[0082] The subject of operation of each of the following flowcharts (electronic musical instrument 10) may be interpreted as either the CPU 201, the waveform data output unit 211 (or the sound source LSI 204 and voice synthesis LSI 205 therein (processing unit 306, singing voice control unit 307, sound source 308, singing voice synthesis unit 309, etc.)), or a combination of these. For example, each operation may be performed by the CPU 201 executing a control processing program loaded from the ROM 202 to the RAM 203.

[0083] Note that an initialization process may be performed at the start of the flow shown below. The initialization process may include interrupt processing, lyric progression, deriving TickTime, which is the reference time for automatic accompaniment, tempo setting, song selection, song loading, instrument sound selection, and other button-related processing.

[0084] The CPU 201 can detect operations of the switch panel 140b, the keyboard 140k, the pedals 140p, etc., based on an interrupt from the key scanner 206 at an appropriate timing, and execute the corresponding processing.

[0085] In the following, an example of controlling the progression of lyrics is shown, but the target of the progression control is not limited to this. For example, based on the present disclosure, the progression of any character string, sentence (e.g., news script), etc. may be controlled instead of lyrics. In other words, the lyrics of the present disclosure may be read as characters, character strings, etc.

[0086] First, an overview of a method for controlling syllable positions in lyrics (which may also be called lyrics, phrases, etc.) in this disclosure will be described. This control method allows lyrics to be controlled quickly and intuitively using a keyboard. In this disclosure, a "syllable" is described as meaning one word (or one character), such as "go," "for," or "it," and a "lyric" or "phrase" is described as meaning a word (or sentence) consisting of multiple syllables or multiple words (or multiple characters), such as "Go for it," but these definitions may be different.

[0087] In addition, in the present disclosure, a syllable position may be represented by a specific index (for example, called a syllable index). The syllable index may be a variable indicating which syllable (or character) from the beginning of the lyrics a syllable index corresponds to. In the present disclosure, the syllable position and the syllable index may be read as interchangeable.

[0088] In the present disclosure, lyrics corresponding to one syllable index may correspond to one or more characters constituting one syllable. The syllable may include various syllables such as only vowels, only consonants, or a consonant plus a vowel.

[0089] 6 is a diagram showing an example of division of a keyboard into key ranges for syllable position control according to an embodiment. In this example, the keyboard 140k is divided into a first key range (first range) and a second key range (second range). Note that, although this example shows an example in which the number of keys in the keyboard 140k is 61, the embodiment of the present disclosure is similarly applicable to other numbers of keys.

[0090] In the present disclosure, the key range may be interpreted interchangeably as the area (or range) of the keyboard, the area (or range) of the performance operators, the range, the sound area (or range), and the like.

[0091] The first key area may be called the syllable position control key area, the keyboard control key area, or simply the control key area, and is used to specify the syllable position. In other words, the control key area does not have to be used to specify the pitch, velocity, duration, etc. of the note to be played.

[0092] As an example, the control range may correspond to a range of keys for chord sounding (e.g., C1-F2). Within the control range, the keys used for controlling syllable positions may be composed of only white keys, only black keys, or both. For example, when only white keys are used for controlling syllable positions, the black keys in the control range may be used for controlling lyrics (e.g., transitions to the next / previous lyrics in a song).

[0093] The second range may be called the keyboard performance range or simply the performance range, and is used to specify pitch, velocity, duration, etc. The electronic musical instrument 10 produces sounds corresponding to syllable positions (or lyrics) specified by operating the control range, using the pitch (interval), velocity, etc. specified by operating the performance range.

[0094] In Fig. 6, the control range is composed of some keys on the left hand side, and the performance range is composed of keys that do not fall under the control range, but this is not limited to the above. For example, each range may be composed of non-adjacent (discrete) keys, or the control range may be composed of keys on the right hand side, and the performance range may be composed of keys on the left hand side.

[0095] 7A-7C are diagrams showing an example of syllables assigned to the control key range. Fig. 7A shows an example of lyrics for which the syllable position is controlled by the control key range. The lyrics shown are "Blink and see everyone." The pitch and duration of the sound are examples, and the actual sound output can be controlled by the performance key range.

[0096] Figure 7B shows an example of assigning each syllable of the lyrics of Figure 7A to a white key in the control key range. In this example, one syllable of the lyrics is mapped to each of the 11 white keys C1-F2 in the control key range.

[0097] When a white key in the control key range is pressed, the electronic musical instrument 10 sets the syllable position to the position corresponding to that white key (for example, if the white key is G1, it sets it to "shi"). When C1 is pressed, the electronic musical instrument 10 starts the lyrics at the beginning regardless of the current syllable position (sets the syllable position to "ma").

[0098] When any key in the performance range is pressed while no key in the control range is pressed, the electronic musical instrument 10 shifts the syllable position by one (for example, if the position before the key pressing was "ma", it shifts to "ba"). When the syllable position reaches the end of the lyrics, the syllable position may be changed to the beginning of the lyrics ("ma" in FIG. 7B) or to the beginning of the lyrics following the current lyrics.

[0099] Even if a white key in the control key range remains pressed and any key in the performance key range is pressed multiple times, the electronic musical instrument 10 maintains the syllable position at the position corresponding to that white key (for example, if the position corresponding to that white key is "shi", then "shi" is pronounced each time a key in the performance key range is pressed).

[0100] When a white key in the control range is pressed, if a key in the performance range is already pressed, the electronic musical instrument 10 may sound a syllable corresponding to the white key based on the pressed key in the performance range. For example, when keys in the performance range are pressed, if keys C2, D1, and E1 are pressed in the control range, the electronic musical instrument 10 may sound "mibata" at the pitch corresponding to the keys in the performance range. This operation allows the syllables of the lyrics corresponding to the control range to be sounded in any order (by freely creating anagrams).

[0101] 7C shows an example in which each syllable of another lyric (English lyric) is assigned to a white key in the control key range. In this example, each syllable of the lyric "holy infant so tender and mild sleep in" is mapped to each of the eleven white keys C1-F2 in the control key range. In this way, syllables of any language may be assigned.

[0102] One key may be assigned one character / one syllable as shown in Figs. 7B and 7C, or multiple characters / multiple syllables.

[0103] The data relating to lyrics and syllables may correspond to the above-mentioned singing voice data 215 (which may be called lyrics data, syllable data, etc.). For example, the electronic musical instrument 10 may store a plurality of lyrics data in a memory, and select one of the lyrics data when a specific function key (e.g., a button, a switch, etc.) is operated.

[0104] <Lyric progression control> FIG. 8 is a diagram illustrating an example of a flowchart of a lyrics progression control method according to an embodiment.

[0105] First, the electronic musical instrument 10 sets the syllable position control flag to "invalid" as an initial value (step S101).

[0106] The electronic musical instrument 10 determines whether or not syllable assignment is necessary (step S102). The electronic musical instrument 10 may determine that syllable assignment is necessary, for example, when a specific function key (e.g., a button, a switch, etc.) of the electronic musical instrument 10 is operated (and lyrics are loaded, etc.).

[0107] If syllable allocation is required (step S102-Yes), the electronic musical instrument 10 performs a syllable allocation process for the control key range (white keys therein) (step S103), and sets the syllable position control flag to "valid" (step S104). The syllable to be allocated may be one selected from multiple lyric data, as described above. The syllable position control flag being "valid" may be referred to as keyboard split being valid.

[0108] If syllable allocation is not required (step S102-No), the control key range is not set and all keys are used for pitch specification (normal performance mode). The syllable position control flag being "invalid" may be referred to as keyboard split being invalid.

[0109] After step S104 or step S102-No, the electronic musical instrument 10 judges whether any keyboard operation has been performed (step S105). If a keyboard operation has been performed (step S105-Yes), the electronic musical instrument 10 acquires information such as which keys have been / are being pressed and which keys have been / are being released (which may be called key pressed / released information) (step S106).

[0110] After step S106, the electronic musical instrument 10 checks whether the above-mentioned syllable position control flag is valid (step S107). If the syllable position control flag is valid (step S107-Yes), the electronic musical instrument 10 performs syllable position control processing (step S108). If not (step S107-No), the electronic musical instrument 10 performs performance control processing (step S109). The syllable position control processing will be described later with reference to FIG. 9, and the performance control processing will be described later with reference to FIG. 10.

[0111] After step S108 or step S109, the electronic musical instrument 10 determines whether or not the playback of lyrics has finished (step S110). If it has finished (step S110-Yes), the electronic musical instrument 10 may end the processing of this flowchart and return to a standby state. If it has not finished (step S110-No), the electronic musical instrument 10 may return to step S102 or step S105. Here, "whether the playback of lyrics has finished" may refer to the playback of one phrase of lyrics, or the playback of the lyrics of the entire song.

[0112] <Syllabic position control> FIG. 9 is a diagram illustrating an example of a flowchart of a syllable position control process according to an embodiment.

[0113] The electronic musical instrument 10 determines whether or not a key press / release operation has been performed in the control key range (step S201). If an operation has been performed in the control key range (step S201-Yes), the electronic musical instrument 10 determines whether or not the operation is a key press operation (step S202).

[0114] If a key is pressed (step S202-Yes), the electronic musical instrument 10 saves (or stores or sets) information about the key pressed by the key pressing as a syllable control key (step S203). The electronic musical instrument 10 also resets (or does not set) a key-release flag (step S204). The key-release flag is reset if any key in the control key range is pressed, and is set otherwise.

[0115] On the other hand, if a key release operation has been performed (step S202-No), the electronic musical instrument 10 determines whether or not the information about the key released by the key release operation is the same as the stored syllable control key (step S205).

[0116] If the information of the released key is the same as the stored syllable control key (step S205-Yes), a key-release flag is set (step S206). Even if the information of the released key is the same as the stored syllable control key, if there is still a pressed key in the control key range, the electronic musical instrument 10 may store the information of the pressed key as a syllable control key, and in this case the key-release flag may not be set.

[0117] On the other hand, if no operation has been performed in the control key range (step S201-No), the electronic musical instrument 10 performs a performance control process (step S207). The performance control process in step S207 may be the same as the performance control process in step S109.

[0118] After step S204, step S206, step S205-No, or step S207, the electronic musical instrument 10 may end the syllable position control process.

[0119] The syllable control key may be called syllable control information, and may be information on the key number of the pressed / released key, or information on the pitch (or note number) of the pressed / released key. In the following, the present disclosure will be described taking as an example a key number being held as the syllable control key, but is not limited to this.

[0120] For example, the keys corresponding to C1-F2 in the examples of Figures 7B and 7C may correspond to key numbers 0 to 11. The key numbers may be character strings representing pitches (for example, C1, F2).

[0121] According to the syllable position control process of Fig. 9, when a key in the control key range is pressed, the key is held. When a key in the control key range is released, the held key is maintained and a key release flag is set. When another key in the control key range is pressed, the held key is replaced with the other key. Note that when a new key is pressed while a key in the control key range is not released, the held key may be overwritten with the new key.

[0122] <Performance Control> FIG. 10 is a diagram illustrating an example of a flowchart of a performance control process according to an embodiment.

[0123] The electronic musical instrument 10 executes a syllable progression determination process (step S301). The syllable progression determination process returns a determination result (return value) regarding whether or not to advance the syllable position. If the determination result is Yes (or True), the current syllable position is acquired, and the syllable position is shifted or advanced by one (in other words, the lyrics are advanced) (step S302). An example of the syllable progression determination process will be described later with reference to FIG. 11.

[0124] On the other hand, if the result of the syllable progression determination process in step S301 is No (or False), the syllable position is not changed.

[0125] After step S302, the electronic musical instrument 10 determines whether or not the syllable control key is set (a valid value is stored) (step S303). If the syllable control key is set (step S303-Yes), the electronic musical instrument 10 determines whether or not the syllable control key is a syllable position designation valid key (which may simply be called a valid key) (step S304).

[0126] Here, the valid keys may refer to all the keys in the control key range to which syllables are assigned. For example, if the number of syllables in the current lyrics is less than the number of white keys in the control key range, some of the white keys in the control key range are valid keys, and the rest are not. In this case, the black keys are also not valid keys.

[0127] As you can see, if the lyrics change, the effective keys may change. Note that one key does not have to correspond one-to-one to one syllable, one key may correspond to multiple syllables, or multiple keys may correspond to one syllable.

[0128] If the syllable control key is a valid key (step S304-Yes), the electronic musical instrument 10 obtains the syllable position corresponding to the syllable control key (its key number) (step S305).

[0129] After step S305, the electronic musical instrument 10 judges whether the key-release flag is set (step S306). If the key-release flag is set (step S306-Yes), the electronic musical instrument 10 clears the syllable control key (or may set an invalid value) (step S307).

[0130] After step S303-No, step S304-No, step S306-No, or step S307, the electronic musical instrument 10 performs a syllable change process (step S308). An example of the syllable change process will be described later with reference to Fig. 12. As will be described later, a syllable performance (playback) process may be performed as part of the syllable change process.

[0131] Incidentally, before or after the syllable change processing, the electronic musical instrument 10 may store the current syllable position (the syllable position acquired (or acquired and advanced by one) in step S302 or step S305) in the storage unit as the current syllable position. The acquisition of the syllable position in step S302 may be acquisition of the current syllable position that is stored. Also, instead of advancing the syllable position by one in step S302, the syllable position may be advanced by one before or after the syllable change processing in step S308.

[0132] After step S301-No or step S308, the electronic musical instrument 10 may end the performance control process.

[0133] <Syllable progression determination> 11 is a diagram showing an example of a flowchart of a syllable progression determination process according to an embodiment. In other words, this process corresponds to a process in which a syllable progresses when a single note is pressed in the performance range, and a process in which a syllable progresses based on which note of the chord (which may be interpreted as "which note" or "which part") has changed due to the key pressing when a chord is pressed in the performance range.

[0134] The electronic musical instrument 10 acquires the current number of pressed keys in the performance key range (step S401).

[0135] Next, the electronic musical instrument 10 judges whether the number of currently pressed keys in the performance range is two or more (whether two or more keys are pressed) (step S402). If the number of currently pressed keys is two or more (step S402-Yes), the electronic musical instrument 10 acquires the key pressing time and key number corresponding to each pressed key (step S403).

[0136] After step S403, the electronic musical instrument 10 judges whether or not the difference between the latest key pressing time and the previous key pressing time in the performance key range is within the chord discrimination time (step S404). Step S404 may be rephrased as, for example, a step of judging whether the difference between the key pressing time of a newly pressed note and the key pressing time of a note pressed the previous time (or i times ago (i is an integer)) is within the chord discrimination time. The previous key pressing time preferably corresponds to a key that has been continuously pressed even in the latest key pressing time.

[0137] Here, the chord discrimination time is a time (period) for determining that a plurality of sounds produced within that time are simultaneous chords, and determining that a plurality of sounds produced outside that time are independent sounds (e.g., sounds of a melody line) or arpeggios. The chord discrimination time may be expressed in milliseconds or microseconds, for example.

[0138] The chord discrimination time may be obtained from a user input or may be derived based on the tempo of the song. The chord discrimination time may be referred to as a predetermined set time, a set time, or the like.

[0139] If the difference between the latest key pressing time and the previous key pressing time is within the chord discrimination time (step S404-Yes), the electronic musical instrument 10 determines that the pressed keys are a simultaneous chord (a chord has been specified), and determines to maintain the syllable (not progress the lyrics), setting the return value of the syllable progression discrimination process to No (or False) (step S405).

[0140] According to the determination in step S404, when multiple keys are pressed with the intention of creating a chord, since it is undesirable for the syllables to advance by the number of keys, the lyrics can be advanced by only one.

[0141] On the other hand, if there is no past key pressing time within the chord discrimination time (step S404-No), it is determined whether the number of currently pressed keys in the performance range is equal to or exceeds a predetermined number and whether the most recently pressed key corresponds to a specific key among all the keys pressed in the performance range (step S406). Note that, if step S404-No, the electronic musical instrument 10 may determine that the chord designation has been released, or that no chord has been designated.

[0142] The predetermined number may be, for example, 2, 4, 8, etc. Also, the specific note (key) may be the lowest note (key) among all the pressed keys, or the i-th (i is an integer) highest or lowest note (key). These predetermined numbers, specific notes, etc. may be set by a user operation, or may be specified in advance.

[0143] If step S406--Yes, the electronic musical instrument 10 determines to advance the syllable (advance the lyrics), and sets the return value of the syllable progression determination process to Yes (or True) (step S407).

[0144] If step S406--No, the electronic musical instrument 10 determines that although there is no simultaneous chord, the syllables are to be maintained (the lyrics are not to be progressed), and sets the return value of the syllable progression determination process to No (or False) (step S405).

[0145] If step S402--No, the electronic musical instrument 10 determines that the chord is not simultaneous and therefore advances the syllable (advances the lyrics), and sets the return value of the syllable progression determination process to Yes (or True) (step S407).

[0146] According to the syllable progression determination process shown in FIG. 11, for example, if there are multiple sounds with a large time difference between their pronunciations (melody), rather than multiple sounds with a small time difference between their pronunciations (so-called simultaneous chords (harmony)), it is possible to cause the syllable to progress.

[0147] <Syllable change> FIG. 12 is a diagram illustrating an example of a flowchart of the syllable changing process according to an embodiment.

[0148] The electronic musical instrument 10 acquires the lyric control data corresponding to the syllable position already acquired in the performance control process (step S501).

[0149] Here, the lyrics control data may be data including parameters related to the pronunciation (vocal synthesis) of each syllable included in the lyrics. If data including parameters related to the pronunciation of a certain syllable is called syllable control data, the lyrics control data may be configured including one or more syllable control data.

[0150] For example, the syllable control data may include information such as pronunciation timing, syllable start frame, vowel start frame, vowel end frame, syllable end frame, lyrics (or syllable) (character information), etc. Note that a frame may be a constituent unit of the above-mentioned phoneme (phoneme string), or may be interpreted as another time unit. In the following description, lyrics control data and syllable control data will not be particularly distinguished from each other.

[0151] The sounding timing may indicate a reference timing (or offset) for each frame (e.g., a syllable start frame, a vowel start frame, etc.). The sounding timing may be given as a time from a key press. The sounding timing and information for each frame may be specified by the number of frames (frame unit).

[0152] The pronunciation of a sound corresponding to a syllable may start from the syllable start frame and end at the syllable end frame. The pronunciation of a sound corresponding to a vowel among the syllables may start from the vowel start frame and end at the vowel end frame. In other words, the vowel start frame usually has a value equal to or greater than the syllable start frame, and the vowel end frame usually has a value equal to or less than the syllable end frame.

[0153] The syllable start frame may correspond to start address information of a syllable frame, and the syllable end frame may correspond to end address information of a syllable frame.

[0154] Next, the electronic musical instrument 10 determines whether the syllable start frame of the lyrics control data obtained in step S501 needs to be adjusted (step S502). For example, if the frame position adjustment flag is set, the electronic musical instrument 10 may determine that the syllable start frame needs to be adjusted. The electronic musical instrument 10 may control the value of the frame position adjustment flag based on the operation of a function key, or may determine the value of the frame position adjustment flag based on a parameter of the lyrics control data.

[0155] If the syllable start frame needs to be adjusted (step S502-Yes), the electronic musical instrument 10 adjusts the syllable start frame based on the adjustment coefficient (step S503). For example, the electronic musical instrument 10 may calculate a new (adjusted) syllable start frame by applying a predetermined operation (e.g., addition, subtraction, multiplication, division) using the adjustment coefficient to the syllable start frame.

[0156] The adjustment factor may be a parameter (e.g., an offset amount, a number of frames, etc.) appropriate for reducing (or removing) the white noise portion of a syllable. The adjustment factor may have a different (or independent) value for each syllable. The adjustment factor may be included in the lyrics control data or may be determined based on the lyrics control data.

[0157] The adjustment of the syllable start frame in step S503 may be applied only to sounds that are generated while a key in the control range is being pressed, or may be applied to sounds that are generated when no key in the control range is being pressed.

[0158] After step S503, the electronic musical instrument 10 determines whether the adjusted syllable start frame value is greater than the vowel start frame value (step S504). If the adjusted syllable start frame value is greater than the vowel start frame value (step S504-Yes), the electronic musical instrument 10 changes the adjusted syllable start frame value to the vowel start frame value (step S505).

[0159] According to steps S504 and S505, for example, pronunciation can be started from the beginning of a vowel while reducing white noise as much as possible. If pronunciation is started halfway through a vowel, the attack of the pronunciation deteriorates, but by starting pronunciation from the beginning of a vowel, the deterioration of the attack can be suppressed.

[0160] After step S502-No, step S504-No or step S505, the electronic musical instrument 10 sets information including at least a syllable start frame, a vowel start frame, a vowel end frame and a syllable end frame as singing voice reproduction information (step S506). The syllable start frame here may be the value of the syllable start frame included in the lyrics control data as described above, or may be the value of the syllable start frame adjusted using an adjustment coefficient, or may be the value of the vowel start frame.

[0161] The electronic musical instrument 10 applies a vocal reproduction process to generate a sound corresponding to the current syllable position (step S507). In the vocal reproduction process, the electronic musical instrument 10 may generate a sound corresponding to the current syllable position based on the vocal reproduction information of step S506 and the key (or the pitch obtained from) pressed in the performance range.

[0162] In the singing voice playback process, the electronic musical instrument 10 may, for example, obtain acoustic feature data (formant information) of the singing voice data corresponding to the current syllable position from the singing voice control unit 307, instruct the sound source 308 to produce an instrument sound of a pitch corresponding to the key pressed (generate instrument sound waveform data), and instruct the singing voice synthesis unit 309 to add the above-mentioned formant information to the instrument sound waveform data output from the sound source 308.

[0163] For example, the processing unit 306 inputs the designated pitch data (pitch data corresponding to the pressed key), singing voice data corresponding to the current syllable position, and singing voice reproduction information corresponding to the current syllable position to the singing voice control unit 307. The singing voice control unit 307 estimates an acoustic feature sequence 317 based on the input, and outputs corresponding formant information 318 and vocal cord sound source data (pitch information) 319 to the singing voice synthesis unit 309. The reproduction start frame of this acoustic feature sequence 317 may be adjusted based on the singing voice reproduction information.

[0164] The singing voice synthesis unit 309 generates singing voice waveform data based on the input formant information 318 and vocal cord sound source data (pitch information) 319, and outputs the data to the sound source 308. Then, the sound source 308 performs sound generation processing on the singing voice waveform data acquired from the singing voice synthesis unit 309.

[0165] In addition, even if the result of the syllable progression determination process in step S301 is No (or False), the electronic musical instrument 10 may sound the sound corresponding to the current syllable position by applying the singing voice reproduction process based on the already obtained singing voice reproduction information and the key pressed in the performance key range.

[0166] <Modification> In the electronic musical instrument 10, at least one of a letter, figure, design, or pattern may be displayed on a key to which a syllable within the control key range is assigned so that the assigned syllable can be visually recognized (or distinguished, grasped, or understood), and at least one of the color, brightness, and saturation of the key (for example, a light-emitting element (light-emitting diode (LED)) built into the key) may be changed.

[0167] In addition, in the electronic musical instrument 10, the key corresponding to the current syllable position may be displayed with at least one of a letter, figure, design, or pattern different from that of the other keys, or at least one of a color, brightness, and saturation of the key different from that of the other keys, so that the key can be visually recognized (or distinguished, grasped, or understood) as the current syllable position (in other words, so that the key can be distinguished from other keys).

[0168] 13A and 13B are diagrams showing an example of the appearance of the keys in the control key range. In this example, the lyrics "Mabataki shite ha minna wo" are visibly displayed on each of the eleven white keys C1-F2 in the control key range.

[0169] Also, in Fig. 13A, part of the C1 key is illuminated (the "circle" in the figure). In Fig. 13B, part of the D1 key is illuminated (the "circle" in the figure). In Figs. 13A and 13B, the player can easily understand that the current syllable position is "ma" and "ba", respectively.

[0170] 13A and 13B, when the keys to which the syllables are assigned are displayed so that the user can easily understand them, the number of keys in the control range does not need to be fixed, and may be variable according to the lyrics currently being played. For example, when the number of syllables in the lyrics is x (x is an integer), the control range only needs to include x white keys. In this case, it is possible to prevent a situation in which the number of keys in the performance range is always small (there is little freedom in the pitches that can be played) regardless of the lyrics selected.

[0171] In the above embodiment, it is assumed that the lyric data is selected based on the operation of a specific function key (e.g., a button, a switch, etc.), but this is not limited to the above. For example, the electronic musical instrument 10 may select lyric data based on the operation of a key (e.g., a black key) in the control key range to which no syllable is assigned. For example, the leftmost black key in the control key range may indicate the selection of the lyric immediately before the current lyric in a piece of music, and the second black key from the left in the control key range may indicate the selection of the lyric immediately after the current lyric in a piece of music.

[0172] The electronic musical instrument 10 may control the display 150d to display lyrics. For example, lyrics near the current lyrics position (syllable index) may be displayed, or lyrics corresponding to sounds being pronounced or sounds that have been pronounced may be displayed in a color that allows the current lyrics position to be identified.

[0173] The electronic musical instrument 10 may transmit at least one of the singing voice data, information on the current lyrics position, etc. to an external device (e.g., a smartphone or a tablet terminal). The external device may control the display of lyrics on its own display based on the received singing voice data, information on the current lyrics position, etc.

[0174] In the above example, the electronic musical instrument 10 is a keyboard-like instrument, but is not limited to this. The electronic musical instrument 10 may be any instrument that has a configuration that allows the user to specify the timing of sound generation, such as an electric violin, an electric guitar, a drum, or a trumpet.

[0175] For this reason, a "key" in the present disclosure may be interpreted as a string, a valve, other performance operators for specifying pitch, any performance operator, etc. A "key press" in the present disclosure may be interpreted as striking a key, picking, playing, operating an operator, user operation, etc. A "key release" in the present disclosure may be interpreted as stopping a string, muting, stopping performance, stopping (not operating) an operator, etc.

[0176] Furthermore, the operators (e.g., performance operators, keys) of the present disclosure may be operators (e.g., images of keys) displayed on a touch panel, a virtual keyboard, etc. In this case, electronic musical instrument 10 is not limited to a so-called musical instrument (e.g., a keyboard), and may be interpreted as a mobile phone, a smartphone, a tablet terminal, a personal computer (PC), a television, etc.

[0177] 14 is a diagram showing an example of a tablet terminal that implements a lyrics progression control method according to an embodiment. The tablet terminal 10t displays at least a keyboard 140k on a display. A part of the keyboard 140k (in this example, a total of 11 white keys from C1 to F2) corresponds to the control key range, and the lyrics "Blink, everyone" are displayed so as to be visible on each of the total of 11 white keys from C1 to F2 in the control key range.

[0178] Furthermore, the external device that has received the singing voice data and information regarding the current lyric position may also display a keyboard 140k indicating the assigned syllables and the current syllable position as shown in FIG.

[0179] As described above, the electronic musical instrument 10 of the present disclosure can provide a new performance experience, allowing the user (player) to enjoy playing even more.

[0180] For example, the electronic musical instrument 10 of the present disclosure can easily locate the beginning of lyrics. Since the position of a syllable can be visually identified, it is possible to jump directly to any syllable during the performance of lyrics.

[0181] Furthermore, when a syllable (vowel) needs to be kept at a specific syllable position during lyrics performance, the electronic musical instrument 10 of the present disclosure can directly specify and maintain any vowel using only the keyboard. Melisma performance is possible without using pedals or buttons.

[0182] In addition, the electronic musical instrument 10 of the present disclosure can randomly change the syllable position according to the operation of the keyboard, and can perform while changing the combination of syllables. Therefore, in addition to the original lyrics, it is possible to create different lyrics like anagrams. For example, when combined with automatic performance such as loop performance or an arpeggiator, it is possible to provide a new performance experience that creates lyric phrases that exceed the user's expectations.

[0183] The electronic musical instrument 10 may include a plurality of performance operators (e.g., keys) each associated with different pitch data, and a processor (e.g., a CPU). The processor may determine the syllable position included in a phrase based on an operation (e.g., key press / key release) on a performance operator included in a first range (control key range) among the plurality of performance operators. The processor may also instruct the pronunciation of a syllable corresponding to the determined syllable position based on an operation on a performance operator included in a second range (performance key range) among the plurality of performance operators. With such a configuration, the part of the lyrics that the user wants to pronounce can be easily specified using only the keyboard, for example.

[0184] In addition, when a performance operator included in the first range is operated, the processor may determine the syllable position based on a key number corresponding to the performance operator included in the first range that is operated. With this configuration, the syllable can be intuitively changed to any syllable by pressing a key in the first range.

[0185] Furthermore, when a performance operator included in the first range is operated and the operated performance operator included in the first range is a valid key to which a syllable is assigned, the processor may determine the syllable position based on a key number corresponding to the operated performance operator included in the first range. With this configuration, a syllable can be intuitively changed to any syllable by operating a key in the first range to which a syllable is assigned. Keys to which no syllable is assigned can be used for purposes other than changing the syllable.

[0186] Furthermore, the processor may shift the syllable position by one position based on the operation of a performance operator included in the second range when the performance operator included in the first range is not operated. This configuration allows for a user-friendly operation in which a syllable is basically advanced only by operating the second range, and a syllable is jumped by operating the first range only when necessary.

[0187] The processor may also instruct pronunciation of a syllable corresponding to the syllable position by adjusting a syllable start frame based on an adjustment coefficient. With this configuration, it is possible to suitably reduce (or remove) a white noise portion of a syllable.

[0188] In addition, when the value of the syllable start frame adjusted based on the adjustment coefficient is greater than the value of the vowel start frame of the syllable, the processor may set the adjusted value of the syllable start frame to be equal to the value of the vowel start frame. With this configuration, it is possible to suppress deterioration of the sense of attack while reducing white noise as much as possible.

[0189] The processor may control the syllables to be generated not to advance when a performance operator in a first range among the plurality of performance operators is being operated, regardless of how a performance operator in a second range among the plurality of performance operators is operated, and may control the syllables to be generated to advance with each operation of a performance operator in the second range if no operation is being performed on any performance operator in the first range. The processor may also instruct the generation of a syllable corresponding to the syllable position at a pitch designated based on the operation of a performance operator in the second range. This configuration makes it easy to maintain the syllables.

[0190] The processor may also control the syllable not to advance from the position of the syllable corresponding to the performance operator in the first range that is being operated, regardless of how the performance operator in the second range is operated, when the operation of the performance operator in the first range is continued. With this configuration, a user-friendly operation is possible in which the syllable advances basically only by operating the second range, and the first range is operated only when necessary to jump syllables.

[0191] Also, each syllable included in the phrase may be assigned to each performance operator included in the first range. With this configuration, the user can easily grasp the current syllable position.

[0192] The processor may use the performance operators included in the first range to determine the position of the syllable when a specific function key is operated by the user, and may use the performance operators included in the first range to specify the pitch of the sound to be generated (normal mode, normal performance operation) when a specific function key is operated by the user. With this configuration, it is possible to appropriately control whether or not to control the progression of lyrics using keyboard splits.

[0193] The processor may also display a display for the performance operators included in the first range to help the user understand the assigned syllables. With this configuration, the user can easily understand the keys that correspond to the syllables that make up the lyrics, and the next user operation can be appropriately prompted.

[0194] The processor may also control the transmission of information to the external device for displaying a display on the external device for the user to understand the syllables assigned to the performance operators included in the first range. With this configuration, the user can easily recognize the keys corresponding to the syllables that make up the lyrics by visually checking the external device, and the next user operation can be appropriately prompted.

[0195] The block diagrams used to explain the above embodiments show functional blocks. These functional blocks (components) are realized by any combination of hardware and / or software. The means for realizing each functional block is not particularly limited. That is, each functional block may be realized by one physically combined device, or may be realized by two or more physically separated devices connected by wire or wirelessly.

[0196] In addition, terms explained in this disclosure and / or terms necessary for understanding this disclosure may be replaced with terms having the same or similar meanings.

[0197] The information, parameters, etc. described in this disclosure may be expressed using absolute values, may be expressed using relative values ​​from a predetermined value, or may be expressed using other corresponding information. Furthermore, the names used for parameters, etc. in this disclosure are not limiting in any respect.

[0198] The information, signals, etc. described in this disclosure may be represented using any of a variety of different technologies. For example, the data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0199] Information, signals, etc. may be input and output via multiple network nodes. Input and output information, signals, etc. may be stored in a specific location (e.g., memory) or may be managed using a table. Input and output information, signals, etc. may be overwritten, updated, or added. Output information, signals, etc. may be deleted. Input information, signals, etc. may be transmitted to another device.

[0200] Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

[0201] Additionally, software, instructions, information, etc. may be transmitted or received over a transmission medium. For example, if the software is transmitted from a website, server, or other remote source using wired and / or wireless technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave, etc.), then these wired and / or wireless technologies are included within the definition of transmission media.

[0202] Each aspect / embodiment described in this disclosure may be used alone, in combination, or switched according to implementation. In addition, the processing procedures, sequences, flow charts, etc. of each aspect / embodiment described in this disclosure may be reordered unless inconsistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0203] As used in this disclosure, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0204] Any reference to an element using a designation such as "first," "second," etc., used in this disclosure does not generally limit the quantity or order of those elements. These designations may be used in this disclosure as a convenient method of distinguishing between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed or that the first element must precede the second element in some way.

[0205] When the terms "include," "including," and variations thereof are used in this disclosure, these terms are intended to be inclusive, similar to the term "comprising." Further, when used in this disclosure, the term "or" is not intended to be an exclusive or.

[0206] In the present disclosure, "A / B" may mean "at least one of A and B."

[0207] In this disclosure, where articles have been added by translation, such as a, an, and the in English, the disclosure may include that the nouns following these articles are in the plural form.

[0208] Although the invention according to the present disclosure has been described in detail above, it is clear to those skilled in the art that the invention according to the present disclosure is not limited to the embodiments described in the present disclosure. The invention according to the present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the invention defined based on the description of the claims. Therefore, the description of the present disclosure is intended to be illustrative and does not have any limiting meaning on the invention according to the present disclosure.

Claims

1. a plurality of performance operators including a first range of white keys to which a plurality of syllables included in a phrase are assigned for each syllable, and a second range of white keys to which a plurality of syllables included in a phrase are assigned, the plurality of performance operators being associated with different pitch data; a processor, the processor comprising: determining a syllable position based on an operation on one of the first performance operators; instructing the pronunciation of a syllable corresponding to the determined syllable position based on an operation on the second performance operator; Electronic musical instrument.

2. The processor, When the first performance operator is operated, the syllable position is determined based on a key number corresponding to the operated first performance operator.

2. The electronic musical instrument according to claim 1.

3. The processor, When the first performance operator is operated, and the operated first performance operator is a valid key to which a syllable is assigned, the syllable position is determined based on a key number corresponding to the operated first performance operator.

3. The electronic musical instrument according to claim 2.

4. The processor, When the first performance operator is not operated, the syllable position is shifted by one based on the operation of the second performance operator.

4. An electronic musical instrument according to claim 1.

5. The processor, indicating a pronunciation in which a syllable start frame of a syllable corresponding to the syllable position is adjusted based on an adjustment coefficient; 5. An electronic musical instrument according to claim 1.

6. Electronic musical instrument computers, determining a syllable position based on an operation on one of a plurality of performance operators including a plurality of first performance operators which are all white keys included in a first range to which a plurality of syllables included in the phrase are assigned for each syllable, and a plurality of second performance operators included in a second range; instructing the pronunciation of a syllable corresponding to the determined syllable position based on an operation on the second performance operator; method.

7. Electronic musical instrument computers, determining a syllable position based on an operation on one of a plurality of performance operators including a plurality of first performance operators which are all white keys included in a first range to which a plurality of syllables included in the phrase are assigned for each syllable, and a plurality of second performance operators included in a second range; instructing the pronunciation of a syllable corresponding to the determined syllable position based on an operation on the second performance operator; program.

Images