Electronic musical instrument, method and storage medium

Abstract

The present invention provides an electronic musical instrument, a method and a storage medium. The electronic musical instrument comprises: a plurality of performance operating members, each of which corresponds to different pitch data; and at least one processor; the at least one processor performs the following processing: judging whether two or more performance operating members are operated within a set time according to a user operation; and instructing to pronounce a song corresponding to the same lyrics at each pitch corresponding to the two or more performance operating members specified according to the user operation if it is judged that two or more performance operating members are operated within the set time.

Description

Technical Field

[0001] The present invention relates to an electronic musical instrument, a method and a storage medium. Background Art

[0002] In recent years, the use of synthesized sounds has been expanding. In such a process, it is preferred to have an electronic musical instrument that can not only automatically play, but also advance the lyrics according to the keys of the user (player), and output the synthesized sounds corresponding to the lyrics, so as to be able to express the synthesized sounds more flexibly.

[0003] For example, Patent Document 1 discloses a technique for advancing lyrics in synchronization with a performance based on a user's operation using a keyboard or the like.

[0004] Patent Document 1: Japanese Patent No. 4735544 Summary of the invention

[0005] However, in the case where a plurality of notes can be pronounced simultaneously using a keyboard or the like, if the lyrics are simply advanced each time a key is pressed, the lyrics may advance too much when a plurality of keys are pressed simultaneously.

[0006] Therefore, an object of the present invention is to provide an electronic musical instrument, method and storage medium capable of appropriately controlling the progression of lyrics related to performance.

[0007] An electronic musical instrument related to a technical solution of the present invention comprises: a plurality of performance operating members, each corresponding to different pitch data; and at least one processor; the at least one processor performs the following processing: judging whether more than two performance operating members are operated within a set time based on user operation; and when it is judged that more than two performance operating members are operated within the set time, instructing the pronunciation of singing corresponding to the same lyrics with each pitch corresponding to the more than two performance operating members specified according to the user operation.

[0008] According to a technical solution of the present invention, the progression of lyrics related to the performance can be appropriately controlled. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 FIG. 1 is a diagram showing an example of the appearance of an electronic musical instrument 10 according to an embodiment.

[0010] Figure 2 This is a diagram showing an example of the hardware configuration of a control system 200 of an electronic musical instrument 10 according to an embodiment.

[0011] Figure 3 This is a diagram showing a configuration example of the sound learning unit 301 according to one embodiment.

[0012] Figure 4 This is a diagram showing an example of the waveform data output unit 302 according to one embodiment.

[0013] Figure 5 This is a diagram showing another example of the waveform data output unit 302 according to one embodiment.

[0014] Figure 6 This is a diagram showing an example of a flowchart of a lyrics advancement control method related to one embodiment.

[0015] Figure 7 This is a diagram showing an example of a flowchart of lyrics progression determination processing based on chord voicing.

[0016] Figure 8 This is a diagram showing an example of the progression of lyrics controlled by the lyrics progression determination process.

[0017] Fig. 9 This is a diagram showing an example of a flowchart of synchronization processing. DETAILED DESCRIPTION

[0018] Singing with two or more notes for a part that was originally composed with one syllable to one note (syllable form) is also called melisma. Melisma can also be called falsetto (Japanese: フェイク, fake), coloratura (こぶし), etc.

[0019] The inventors of the present invention focused on the fact that when performing the flowery singing method on an electronic musical instrument equipped with a vocal synthesis sound source, the characteristic of flowery singing is to maintain the immediately preceding vowel while freely changing the pitch, and came up with the lyrics advancement control method of the present invention.

[0020] According to a technical solution of the present invention, it is possible to control so that the lyrics are not advanced in the flowery singing. In addition, even when pressing a plurality of keys simultaneously, it is also possible to appropriately control the advancement of the lyrics.

[0021] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description, the same reference numerals are given to the same parts. The same parts have the same names, functions, etc., so the detailed description will not be repeated.

[0022] In addition, in the present invention, "advance of lyrics", "advance of the position of lyrics", "advance of the singing position", etc. can also be replaced with each other. In addition, in the present invention, "not advancing lyrics", "not controlling the advancement of lyrics", "keeping lyrics", "pausing lyrics", etc. can also be replaced with each other.

[0023] (Electronic Musical Instruments)

[0024] Figure 1140 is a diagram showing an example of the appearance of an electronic musical instrument 10 according to an embodiment. The electronic musical instrument 10 may be equipped with a switch (button) panel 140b, a keyboard 140k, a pedal 140p, a display 150d, a speaker 150s, and the like.

[0025] The electronic musical instrument 10 is a device that receives input from the user via operating elements such as a keyboard and a switch to control the performance, the progression of lyrics, etc. The electronic musical instrument 10 may also be a device having a function of generating sounds corresponding to performance information such as MIDI (Musical Instrument Digital Interface) data. The device may also be an electronic musical instrument (electronic piano, synthesizer, etc.), or may be an analog musical instrument equipped with sensors and the like and configured to have the functions of the above-mentioned operating elements.

[0026] The switch panel 140b may also include switches for specifying volume, setting sound source, timbre, etc., selecting songs (accompaniment), starting / stopping song reproduction, setting song reproduction (beat, etc.), etc.

[0027] The keyboard 140k has a plurality of keys as performance operating elements. The pedal 140p may be a sustain pedal that has the function of prolonging the sound of a depressed keyboard while the pedal is depressed, or may be a pedal for operating an effector that processes the timbre, volume, and the like.

[0028] In addition, in the present invention, the damper pedal, pedal, foot switch, controller (operating member), switch, button, touch panel, etc. can also be replaced with each other. The stepping of the pedal in the present invention can also be replaced by the operation of the controller.

[0029] The keys may also be referred to as playing operators, pitch operators, tone operators, direct operators, first operators, etc. The pedals may also be referred to as non-playing operators, non-pitch operators, non-tone operators, indirect operators, second operators, etc.

[0030] The display 150d may also display lyrics, musical scores, various setting information, etc. The speaker 150s may also be used to emit sounds generated by the performance.

[0031] Furthermore, the electronic musical instrument 10 is capable of generating or converting at least one of a MIDI message (event) and an Open Sound Control (OSC) message.

[0032] The electronic musical instrument 10 may also be referred to as a control device 10 , a lyrics advancement control device 10 , or the like.

[0033] The electronic musical instrument 10 may communicate with a network (such as the Internet) via at least one of wired and wireless (for example, Long Term Evolution (LTE), 5th generation mobile communication system New Radio (5G NR), Wi-Fi (registered trademark), etc.).

[0034] The electronic musical instrument 10 may store singing data (also referred to as lyrics text data, lyrics information, etc.) related to the lyrics of the song to be controlled, or transmit and / or receive the singing data via a network. The singing data may be a text recorded in a musical notation language (e.g., MusicXML), may be expressed in a MIDI data storage format (e.g., Standard MIDI File (SMF) format), or may be a text given in a normal text file.

[0035] Alternatively, the electronic musical instrument 10 may acquire content of a user singing in real time via a microphone or the like included in the electronic musical instrument 10 , and acquire text data obtained by applying voice recognition processing thereto as singing voice data.

[0036] Figure 2 This is a diagram showing an example of the hardware configuration of a control system 200 of an electronic musical instrument 10 according to an embodiment.

[0037] Central Processing Unit (CPU) 201, ROM (Read Only Memory) 202, RAM (Random Access Memory) 203, waveform data output unit 211, connected Figure 1 The switch (button) panel 140b, the keyboard 140k, the key scanner 206 of the pedal 140p, and the Figure 1 An LCD controller 208 of an LCD (Liquid Crystal Display) which is an example of the display 150 d is connected to the system bus 209 .

[0038] A timer 210 for controlling the sequence of the automatic performance may also be connected to the CPU 201. The CPU 201 may also be called a processor, and may include an interface with peripheral circuits, a control circuit, a calculation circuit, a register, and the like.

[0039] The functions of each device can also be realized by reading the specified software (program) into hardware such as processor 1001 and memory 1002, and the processor 1001 performs calculations and controls communication based on communication device 1004, reading and / or writing of data in memory 1002 and storage device 1003, etc.

[0040] The CPU 201 executes the control program stored in the ROM 202 while using the RAM 203 as a working memory. Figure 1 The control operation of the electronic musical instrument 10. In addition, the ROM 202 may store vocal data, accompaniment data, music (song) data including these, and the like in addition to the control program and various fixed data.

[0041] The CPU 201 is provided with a timer 210 used in the present embodiment, which counts, for example, the progress of the automatic performance of the electronic musical instrument 10 .

[0042] The waveform data output unit 211 may include a sound source LSI (Large Scale Integrated Circuit) 204, a sound synthesis LSI 205, etc. The sound source LSI 204 and the sound synthesis LSI 205 may be integrated into one LSI.

[0043] The singing voice waveform data 217 and the song waveform data 218 outputted from the waveform data output unit 211 are respectively converted into an analog singing voice sound output signal and an analog musical sound output signal by the D / A converters 212 and 213. The analog musical sound output signal and the analog singing voice sound output signal may also be mixed by the mixer 214, and the mixed signal thereof is amplified by the amplifier 215 and then outputted from the speaker 150s or the output terminal.

[0044] The key scanner (scanner) 206 scans stably Figure 1 The key (note on) / key release (note off) status of the keyboard 140k, the switch operation status of the switch panel 140b, the pedal operation status of the pedal 140p, etc., are interrupted to the CPU201 to transmit the state change.

[0045] The LCD controller 208 is an IC (Integrated Circuit) that controls the display state of an LCD as an example of the display 150 d .

[0046] In addition, the system structure is an example and is not limited to this. For example, the number of circuits included is not limited to this. The electronic musical instrument 10 may have a structure that does not include a part of the circuit (mechanism), or may have a structure in which the function of one circuit is realized by multiple circuits. It may also have a structure in which the functions of multiple circuits are realized by one circuit.

[0047] In addition, the electronic musical instrument 10 may also include hardware such as a microprocessor, a digital signal processor (DSP), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array), and may also implement a part or all of each functional block by the hardware. For example, the CPU 201 may also be implemented by at least one of these hardware.

[0048] <Generation of Acoustic Model>

[0049] Figure 3 1 is a diagram showing an example of a configuration of a voice learning unit 301 according to an embodiment. The voice learning unit 301 may also be a unit composed of Figure 1 The sound learning unit 301 may be installed as a function executed by the server computer 300 which is separated from the electronic musical instrument 10 and exists outside. In addition, the sound learning unit 301 may be built into the electronic musical instrument 10 as a function executed by the CPU 201, the sound synthesis LSI 205, etc.

[0050] The speech learning unit 301 and the waveform data output unit 302 described later for realizing speech synthesis of the present invention may be implemented based on, for example, a statistical speech synthesis technique based on deep learning.

[0051] The speech learning unit 301 may include a training text analyzing unit 303 , a training acoustic feature extraction unit 304 , and a model learning unit 305 .

[0052] The voice learning unit 301 uses, for example, data obtained by recording a singer singing a plurality of songs of an appropriate genre as training singing voice data 312. In addition, lyrics texts of the songs are prepared as training singing voice data 311.

[0053] The training text analysis unit 303 takes the training singing data 311 including the lyrics text as input and analyzes the data. As a result, the training text analysis unit 303 estimates and outputs a training language feature value sequence 313, which is a discrete numerical sequence representing the phonemes, pitches, etc. corresponding to the training singing data 311.

[0054] The training sound feature extraction unit 304 analyzes the training singing voice sound data 312 as input, which is data recorded by a singer singing lyrics text corresponding to the training singing voice data 311 in accordance with the input of the training singing voice data 311, etc., through a microphone, etc. As a result, the training sound feature extraction unit 304 extracts and outputs a training sound feature sequence 314 representing the features of the sound corresponding to the training singing voice sound data 312.

[0055] In the present invention, the training acoustic feature sequence 314 and the acoustic feature sequence corresponding to the acoustic feature sequence 317 described later include acoustic feature data (which may also be referred to as formant information, spectral information, etc.) that models the human vocal tract, and vocal cord sound source data (which may also be referred to as sound source information) that models the human vocal cords. As spectral information, for example, Mel frequency cepstrum, line spectral pairs (LSP), etc. may be used. As sound source information, the fundamental frequency (F0) and power value that represent the pitch frequency of the human voice may be used.

[0056] The model learning unit 305 estimates an acoustic model that maximizes the probability of generating a training acoustic feature sequence 314 based on the training language feature sequence 313 by machine learning. That is, the relationship between the language feature sequence as text and the acoustic feature sequence as sound is expressed by a statistical model such as an acoustic model. The model learning unit 305 outputs the model parameters of the acoustic model calculated as a result of machine learning as a learning result 315. Therefore, this acoustic model corresponds to a trained model.

[0057] As the acoustic model represented by the learning result 315 (model parameter), HMM (Hidden Markov Model) may be used.

[0058] When a singer sings lyrics that follow a certain melody, the HMM acoustic model can be used to learn how the characteristic parameters of the singing voice, such as the vibration of the vocal cords and the characteristics of the vocal tract, change over time. More specifically, the HMM acoustic model can be a model obtained by modeling the spectrum, fundamental frequency, and their time structure obtained from the singing voice data for training in phoneme units.

[0059] First, the HMM acoustic model is used Figure 3The model learning unit 305 in the sound learning unit 301 may also learn the HMM sound model with the maximum likelihood by inputting the training language feature sequence 313 output by the training text analysis unit 303 and the training sound feature sequence 314 output by the training sound feature extraction unit 304.

[0060] The spectral parameters of singing voice can be modeled by continuous HMM. On the other hand, since the logarithmic fundamental frequency (F0) is a time series signal of variable dimensions that takes continuous values ​​in the voiced interval and has no value in the silent interval, it cannot be directly modeled by the usual continuous HMM or discrete HMM. Therefore, an HMM based on probability distribution in multiple spaces corresponding to variable dimensions, namely MSD-HMM (Multi-Space probability Distribution HMM: multi-space probability distribution hidden Markov model), is used. The Mel-frequency cepstrum is used as a multi-dimensional Gaussian distribution as the spectral parameter, and the voiced logarithmic fundamental frequency (F0) is modeled as a Gaussian distribution in a 1-dimensional space and the silent space is modeled as a Gaussian distribution in a 0-dimensional space at the same time.

[0061] In addition, it is known that even if the sound characteristics of the phonemes are the same, the characteristics of the phonemes constituting the singing voice are affected by various factors and change. For example, the spectrum and logarithmic fundamental frequency (F0) of the phoneme, which is the basic phonological unit, vary depending on the singing style or tempo, or the lyrics or pitch before and after. Such factors that affect the sound characteristics are called context.

[0062] In the statistical sound synthesis processing of one embodiment, in order to model the acoustic features of the sound with high accuracy, an HMM acoustic model (context-dependent model) that takes context into consideration may also be used. Specifically, the training text analysis unit 303 may also output a training language feature quantity sequence 313 that takes into consideration not only the phonemes and pitches of each frame, but also the previous and next phonemes, the current position, the previous and next vibrato, intonation, etc. Furthermore, in order to make the combination of contexts effective, context clustering based on a decision tree may also be used.

[0063] For example, the model learning unit 305 may also generate a state duration decision tree for determining the state duration based on a training language feature sequence 313 corresponding to the context of many phonemes related to the state duration extracted from the training singing data 311 by the training text analysis unit 303, as a learning result 315.

[0064] In addition, the model learning unit 305 can also generate a mel-cepstrum parameter decision tree for determining mel-cepstrum parameters as a learning result 315, for example, based on a training sound feature sequence 314 corresponding to a plurality of phonemes regarding mel-cepstrum parameters extracted from the training singing sound data 312 by the training sound feature extraction unit 304.

[0065] In addition, the model learning unit 305 may generate a logarithmic fundamental frequency decision tree for determining the logarithmic fundamental frequency (F0) as a learning result 315, for example, based on the training acoustic feature quantity sequence 314 corresponding to a plurality of phonemes related to the logarithmic fundamental frequency (F0) extracted from the training singing voice sound data 312 by the training acoustic feature quantity extraction unit 304. In addition, the voiced interval and the unvoiced interval of the logarithmic fundamental frequency (F0) may be modeled as 1-dimensional and 0-dimensional Gaussian distributions, respectively, by the MSD-HMM corresponding to the variable dimension, and the logarithmic fundamental frequency decision tree may be generated.

[0066] Alternatively, an acoustic model based on a deep neural network (DNN) may be used in place of or in combination with an acoustic model based on an HMM. In this case, the model learning unit 305 may generate model parameters representing a nonlinear transformation function of each neuron in the DNN from the language feature quantity to the acoustic feature quantity as a learning result 315. According to the DNN, it is possible to express the relationship between the language feature quantity sequence and the acoustic feature quantity sequence using a complex nonlinear transformation function that is difficult to express using a decision tree.

[0067] In addition, the acoustic model of the present invention is not limited to these, and for example, an acoustic model combining HMM and DNN may be used. As long as it is a technology that utilizes statistical sound synthesis processing, any sound synthesis method may be used.

[0068] The learning result 315 (model parameters) is, for example, Figure 3 As shown, in Figure 1 When the electronic musical instrument 10 is shipped from the factory, the Figure 2 In the ROM 202 of the control system of the electronic musical instrument 10, when the power of the electronic musical instrument 10 is turned on, Figure 2 The ROM 202 is loaded into the singing voice control unit 306 described later in the waveform data output unit 211.

[0069] The learning result 315 is, for example, Figure 3 As shown, the player may operate the switch panel 140 b of the electronic musical instrument 10 to download the data from the outside such as the Internet via the network interface 219 to the singing voice control unit 306 in the waveform data output unit 211 .

[0070] <Sound synthesis based on acoustic model>

[0071] Figure 4 This is a diagram showing an example of the waveform data output unit 302 according to one embodiment.

[0072] The waveform data output unit 302 includes a processing unit (also called a text processing unit, a pre-processing unit, etc.) 306, a singing control unit (also called an acoustic model unit) 307, a sound source 308, a singing synthesis unit (also called a vocalization model unit) 309, etc.

[0073] The waveform data output unit 302 receives the input based on Figure 1 The 140k keys of the keyboard are connected via Figure 2 The key scanner 206 synthesizes and outputs singing waveform data 217 corresponding to the lyrics and pitch from the singing data 215 including the information of the lyrics and the pitch instructed by the CPU 201. In other words, the waveform data output unit 302 performs statistical sound synthesis processing for synthesizing the singing waveform data 217 corresponding to the singing data 215 including the lyrics text by predicting using a statistical model such as the acoustic model set in the singing control unit 306.

[0074] Furthermore, when the song data is reproduced, the waveform data output unit 302 outputs the song waveform data 218 corresponding to the corresponding song reproduction position.

[0075] The processing unit 307 receives, for example, a result of a player's performance matching the automatic performance, which includes the sound from the player. Figure 2 The CPU 201 may analyze the singing voice data 215 of the information related to the phonemes and pitch of the lyrics specified by the CPU 201. The singing voice data 215 may also include data (such as pitch and note length data) of the nth note (also referred to as the nth note), singing voice data of the nth note, etc.

[0076] For example, the processing unit 307 may determine whether the lyrics are advanced based on the note on / note off data and pedal on / off data obtained by the operation of the keyboard 140k and the pedal 140p, and obtain the singing voice data 215 corresponding to the lyrics to be output based on the lyrics advancement control method described later. In addition, the processing unit 307 may analyze the language feature quantity sequence 316 representing the phonemes, word classes, words, etc. corresponding to the pitch data specified by the key and the acquired singing voice data 215, and output it to the singing voice control unit 306.

[0077] The singing data may also include information including at least one of the lyrics (characters), syllable type (starting syllable, middle syllable, ending syllable, etc.), lyrics index, corresponding pitch (correct pitch), and corresponding pronunciation period (for example, pronunciation start timing, pronunciation end timing, pronunciation length (duration)) (correct pronunciation period).

[0078] For example, in Figure 4 In the example, the singing voice data 215 may include singing voice data of the nth lyrics corresponding to the nth (n=1, 2, 3, 4, ...) note and information on the specified timing (nth singing voice reproduction position) at which the nth note should be reproduced.

[0079] The vocal data 215 may also include information (data in a specific sound file format, MIDI data, etc.) for playing the accompaniment (song data) corresponding to the lyrics. When the vocal data is represented in the SMF format, the vocal data 215 may also include a track block storing data about the vocals and a track block storing data about the accompaniment. The vocal data 215 may also be read from the ROM 202 into the RAM 203. The vocal data 215 may also be stored in a memory (e.g., the ROM 202, the RAM 203) before playing.

[0080] In addition, the electronic musical instrument 10 can also control the progress of automatic accompaniment based on events represented by the singing data 215 (for example, meta-events (timing information) indicating the vocalization timing and pitch of lyrics, MIDI events indicating the pressing or releasing of keys, or meta-events indicating beats, etc.).

[0081] The singing voice control unit 306 estimates the corresponding sound feature sequence 317 based on the language feature sequence 316 input from the processing unit 307 and the sound model set as the learning result 315, and outputs the resonance peak information 318 corresponding to the estimated sound feature sequence 317 to the singing voice synthesis unit 309.

[0082] For example, when the HMM sound model is used, the singing voice control unit 306 refers to the decision tree for each context obtained from the language feature sequence 316, connects the HMMs, and predicts the sound feature sequence 317 (resonance peak information 318 and vocal sound source data 319) with the highest output probability based on the connected HMMs.

[0083] When the DNN acoustic model is used, the singing voice control unit 306 may output the acoustic feature value sequence 317 in the frame unit for the phoneme sequence of the language feature value sequence 316 input in the frame unit.

[0084] exist Figure 4In the process, the processing unit 307 obtains the musical instrument sound data (pitch information) corresponding to the pitch of the pressed sound from the memory (which may be either the ROM 202 or the RAM 203 ), and outputs it to the sound source 308 .

[0085] The sound source 308 generates a sound source signal (also referred to as musical instrument sound waveform data) of musical instrument sound data (pitch information) corresponding to the sound to be produced (the sound of the pressed key) based on the key pressing / release data input from the processing unit 307, and outputs it to the singing voice synthesis unit 309. The sound source 308 may also perform control processing such as envelope control of the sound to be produced.

[0086] The singing voice synthesis unit 309 forms a digital filter that models the vocal tract based on the sequence of formant information 318 sequentially input from the singing voice control unit 306. In addition, the singing voice synthesis unit 309 uses the sound source signal input from the sound source 309 as an excitation source signal and applies the digital filter to generate and output the singing voice waveform data 217 of the digital signal. In this case, the singing voice synthesis unit 309 can also be called a synthesis filter unit.

[0087] In addition, the singing voice synthesis unit 309 may adopt various voice synthesis methods represented by the cepstrum voice synthesis method and the LSP voice synthesis method.

[0088] exist Figure 4 In the example, since the output singing waveform data 217 uses the instrument sound as the sound source signal, the realism is slightly lost compared to the singer's singing voice, but it is a singing voice that retains both the atmosphere of the instrument sound and the sound quality of the singer's singing voice well, and effective singing waveform data 217 can be output.

[0089] In addition, the sound source 309 may also operate together with the processing of the instrument sound waveform data to output the output of other channels as the song waveform data 218. This makes it possible to produce accompaniment sounds as normal instrument sounds, or to produce the instrument sound of the melody line and the singing voice of the melody at the same time.

[0090] Figure 5 FIG. 1 is a diagram showing another example of the waveform data output unit 302 according to one embodiment. Figure 4 Duplicate content will not be described again.

[0091] Figure 5The singing voice control unit 306 estimates the acoustic feature sequence 317 based on the acoustic model as described above. Furthermore, the singing voice control unit 306 outputs the formant information 318 corresponding to the estimated acoustic feature sequence 317 and the vocal cord sound source data (pitch information) 319 corresponding to the estimated acoustic feature sequence 317 to the singing voice synthesis unit 309. The singing voice control unit 306 may estimate the estimated value of the acoustic feature sequence 317 such that the probability of generating the acoustic feature sequence 317 is maximized.

[0092] The singing voice synthesis unit 309 can also generate the following data (for example, it can also be called singing voice waveform data of the nth lyric corresponding to the nth note) and output it to the sound source 308. The data is used to generate a signal of a digital filter that models the vocal tract based on a sequence of resonance peak information 318, which is a pulse sequence that periodically repeats the fundamental frequency (F0) and power value contained in the vocal cord sound source data 319 input from the singing voice control unit 306 (in the case of sound phonemes), or white noise with the power value contained in the vocal cord sound source data 319 (in the case of no sound phonemes), or a mixed signal thereof.

[0093] The sound source 308 generates and outputs the singing voice waveform data 217 as a digital signal based on the singing voice waveform data of the nth lyrics corresponding to the sound (of the key press) to be pronounced, based on the key press / release data input from the processing unit 307 .

[0094] exist Figure 5 In the example, since the output singing waveform data 217 uses the sound generated by the sound source 308 based on the vocal cord sound source data 319 as the sound source signal, it is a signal completely modeled by the singing control unit 306, and can output the singing waveform data 217 of the singing voice that is very realistic and natural to the singer's singing voice.

[0095] Thus, the sound synthesis of the present invention is different from the existing vocoder (a method of inputting human speech through a microphone, replacing it with musical instrument sound and synthesizing it). Even if the user (performer) does not sing (in other words, even if the sound signal of the user's real-time pronunciation is not input into the electronic musical instrument 10), the synthesized sound can be output through keyboard operation.

[0096] As described above, by adopting the technology of statistical sound synthesis processing as the sound synthesis method, it is possible to achieve a significantly smaller memory capacity than the conventional segment synthesis method. For example, in an electronic musical instrument of the segment synthesis method, a memory with a storage capacity of several hundred MB is required for the sound segment data, but in the present embodiment, a memory with a storage capacity of only a few MB is sufficient to store the model parameters of the learning result 315. Therefore, a lower-priced electronic musical instrument can be achieved, and a wider range of users can use the high-quality singing performance system.

[0097] Furthermore, in the conventional segment data method, it is necessary to manually adjust the segment data, so a huge amount of time (years) and labor are required in the production of data for singing performance, but in the production of model parameters for the learning result 315 of the HMM acoustic model or the DNN acoustic model in this embodiment, data adjustment is almost unnecessary, so a fraction of the production time and labor is sufficient. Thus, a lower-priced electronic musical instrument can be realized.

[0098] In addition, ordinary users can also use the learning function built into the server computer 300, the voice synthesis LSI 205, etc. that can be used as a cloud service to learn their own voice, the voice of their family members, or the voice of celebrities, etc., and use them as model voices to sing with electronic musical instruments. In this case, it is also possible to achieve a more natural and high-quality singing performance than before as a lower-priced electronic musical instrument.

[0099] (Lyrics forward control method)

[0100] The following describes a lyrics forward control method according to an embodiment of the present invention. Each lyrics forward control method can also be used by the processing unit 307 of the electronic musical instrument 10 described above.

[0101] The action subject (electronic musical instrument 10) in each of the following flowcharts may be replaced by any one of the CPU 201 and the waveform data output unit 211 (or its internal sound source LSI 204 and sound synthesis LSI 205) or a combination thereof. For example, each action may be performed by the CPU 201 executing a control processing program loaded from the ROM 202 to the RAM 203.

[0102] In addition, at the beginning of the process shown below, initialization processing may also be performed. This initialization processing may also include interrupt processing, exporting TickTime as a reference time for lyrics advancement, automatic accompaniment, etc., beat setting, song selection, song reading, instrument sound selection, and processing associated with other buttons, etc.

[0103] The CPU 201 can detect the operation of the switch panel 140 b , the keyboard 140 k , the pedal 140 p , etc. at appropriate timing based on the interrupt from the key scanner 206 , and perform corresponding processing.

[0104] In addition, the following is an example of controlling the forward movement of lyrics, but the object of the forward movement control is not limited to this. Based on the present invention, for example, the forward movement of any character string, article (for example, a news script), etc. can also be controlled instead of the lyrics. That is, the lyrics of the present invention can also be replaced with characters, character strings, etc.

[0105] Figure 6 1 is a diagram showing an example of a flowchart of a lyrics forwarding control method according to an embodiment. Figure 4 example, but can also be based on Figure 5 .

[0106] First, the electronic musical instrument 10 substitutes 0 into the lyrics index (also indicated as "n") indicating the current position of the lyrics and the note number (also indicated as "SKO") indicating the highest note of the key pressed (step S101). In addition, when the lyrics are to be played from the middle (for example, from the last stored position), a value other than 0 may be substituted into n.

[0107] The lyrics index may also be a variable indicating the syllable (or character) corresponding to the first syllable (or character) when the lyrics are considered as a string. Figure 4 , Figure 5 In the present invention, the lyrics corresponding to the position of one lyric (lyric index) may also correspond to one or more characters constituting one syllable. The syllables included in the singing voice data may also include various syllables such as only vowels, only consonants, and consonants + vowels.

[0108] Step S101 may be executed upon the start of a performance (for example, the start of playback of song data), the reading of vocal data, or the like.

[0109] The electronic musical instrument 10 may also reproduce song data (accompaniment) corresponding to the lyrics according to the user's operation (step S102). The user can perform key operations in accordance with the accompaniment to advance the lyrics and perform the music.

[0110] The electronic musical instrument 10 determines whether the reproduction of the song data started in step S102 has been completed (step S103). If it has been completed (step S103 - Yes), the electronic musical instrument 10 may end the process of this flowchart and return to the standby state.

[0111] In this case, the electronic musical instrument 10 may read the singing voice data designated by the user's operation as the forward control object in step S102, and determine whether all the singing voice data have been forwarded in step S103.

[0112] When the reproduction of the song data has not ended (step S103-No), the electronic musical instrument 10 determines whether a new key has been pressed (a key event has occurred) (step S111). When a new key has been pressed (step S111-Yes), the electronic musical instrument 10 performs lyrics advance determination processing (processing for determining whether to advance the lyrics) (step S112). An example of this processing will be described later. Furthermore, the electronic musical instrument 10 determines whether the lyrics have been advanced as a result of the lyrics advance determination processing (whether it is determined to advance the lyrics) (step S113).

[0113] If it is determined that the lyrics should be advanced (step S113 - yes), the electronic musical instrument 10 increments the lyrics index n (step S114). This increment is basically incremented by 1 (substituting n+1 into n), but a value greater than 1 may be added depending on the result of the lyrics advancement determination process in step S112.

[0114] After the lyrics index is incremented, the electronic musical instrument 10 obtains the acoustic feature data (formant information) of the n-th singing voice data from the singing voice control unit 306 (step S115 ).

[0115] On the other hand, when it is not determined that the lyrics should be advanced (step S113 —No), the electronic musical instrument 10 does not change the lyrics index (maintains the value of the lyrics index). In this case, step S115 is unnecessary, so the process can be simplified.

[0116] After step S115 or S113-No, the electronic musical instrument 10 instructs the sound source 309 to produce the musical instrument sound of the pitch corresponding to the key (generate musical instrument sound waveform data) (step S116). In addition, the electronic musical instrument 10 instructs the singing voice synthesis unit 309 to add the formant information of the n-th singing voice data to the musical instrument sound waveform data output from the sound source 308 (step S117).

[0117] The electronic musical instrument 10 may not advance the lyrics but continue to output the same sound (or the vowels of the same sound) for the sound that is already being pronounced, or may output the sound based on the lyrics that have been advanced. In addition, when the electronic musical instrument 10 emits a sound corresponding to the same value of the lyrics index as the sound that is already being pronounced, it may also emit the vowels of the lyrics and output the sound. For example, when the lyrics such as "Sle" are already being pronounced and the same lyrics are newly pronounced, the electronic musical instrument 10 may also newly emit the sound "e".

[0118] In addition, when there is no new key press (step S111-No), the electronic musical instrument 10 determines whether there is a new key release (a key release event occurs) (step S121). In the case of a new key release (step S121-Yes), the electronic musical instrument 10 performs a muting process on the corresponding singing voice data (step S122). In addition, the electronic musical instrument 10 updates the note management table in sound (step S123).

[0119] Here, the note management table may also manage the note number of the key being sounded (being pressed) and the time when the key is started. In step S123, the electronic musical instrument 10 may also delete the information related to the muted note from the note management table.

[0120] Furthermore, the electronic musical instrument 10 substitutes the note number of the highest tone being produced into SKO (step S124 ).

[0121] Next, the electronic musical instrument 10 determines whether all the keys are released (step S125). If all the keys are released (step S125 - Yes), the electronic musical instrument 10 performs synchronization processing of the lyrics and the song (accompaniment) (step S126). The synchronization processing will be described later.

[0122] After steps S117, S125-No and S126, the process returns to step S103.

[0123] In addition, the electronic musical instrument 10 of the present invention can also use synthesized sounds of different timbres to produce each sound when emitting multiple sounds at the same time. For example, when the user presses four notes, the electronic musical instrument 10 can synthesize and output sounds in a manner corresponding to the timbres of soprano, alto, tenor, and bass in order from the highest note.

[0124] <Lyrics Progress Determination Processing>

[0125] The lyrics progress determination process in step S102 will be described in detail below.

[0126] Figure 7 This is a diagram showing an example of a flowchart of a process for determining the progression of lyrics based on chord voice positions. In other words, this process is equivalent to determining the progression of lyrics based on which pitch (which pitch, which part, etc.) of the chord has changed due to a key press.

[0127] The electronic musical instrument 10 updates the note management table in progress (step S112-1). Here, information about the note of the newly pressed key is added to the note management table. The key press time corresponding to the new key press in step S111 may also be referred to as the current key press time, the latest key press time, etc.

[0128] The electronic musical instrument 10 determines whether the newly pressed sound is higher than SKO (step S112-2). If the newly pressed sound is higher than SKO (step S112-2-Yes), the electronic musical instrument 10 substitutes the note number of the newly pressed sound into SKO and updates SKO (step S112-3). In addition, the electronic musical instrument 10 determines that there are lyrics advancing (step S112-11). This is in consideration of the fact that the highest note (soprano part) is usually equivalent to the melody.

[0129] When the newly pressed sound is not higher than SKO (step S112-2-No), the electronic musical instrument 10 determines whether the difference between the latest pressing time and the previous pressing time is within the chord discrimination time (step S112-4). In other words, step S112-4 may be, for example, a step of determining whether the difference between the pressing time of the newly pressed sound and the pressing time of the sound pressed last (or i times before (i is an integer)) is within the chord discrimination time. The past pressing time preferably corresponds to a key that was continuously pressed during the latest pressing time.

[0130] That is, at least one processor 201 determines whether more than two performance operating members are operated within a set time according to user operation, and when it is determined that more than two performance operating members are operated within the set time, instructs to pronounce the singing voice corresponding to the same lyrics at each pitch corresponding to the above-mentioned more than two performance operating members specified by the user operation. On the other hand, when it is determined that more than two performance operating members are not operated within the set time, instructs to pronounce the singing voice corresponding to the corresponding lyrics at the pitch specified by the user operation. That is, at least one processor determines whether the lyrics are advanced or maintained according to the user operation on any one of the above-mentioned multiple performance operating members.

[0131] Here, the chord determination time is a time (period) for determining multiple sounds emitted within this time as simultaneous chords, and for determining multiple sounds emitted outside this time as independent sounds (for example, sounds of a melody line) or scattered chords. The chord determination time can also be expressed in milliseconds or microseconds, for example.

[0132] The chord identification time may be obtained from a user input or derived based on the tempo of the music. The chord identification time may also be referred to as a predetermined setting time, a setting time, or the like.

[0133] When the difference between the latest key press time and the previous key press time is within the chord identification time (step S112-4-Yes), the electronic musical instrument 10 determines that the key pressed note is a simultaneous chord (a chord is specified) and determines that the lyrics are maintained (the lyrics are not advanced) (step S112-12).

[0134] In addition, when there is no past key-pressing time within the chord discrimination time (step S112-4-No), it is determined whether the current number of key-pressing is greater than a specified number and whether the sound of the newly-pressed key is equivalent to a specific sound among the full tones of the pressed key (step S112-5). In addition, in the case of step S112-4-No, the electronic musical instrument 10 may determine that the designation of the chord is released or that the chord is not designated.

[0135] In addition, the current number of keys can also be determined based on the number of notes in the note management table. In addition, the specified number can also be, for example, 4 notes (assuming 4 notes of soprano, alto, tenor, and bass), or 8 notes. In addition, the specific note can also be the lowest note in the full tone of the key (equivalent to the bass part), or the i-th (i is an integer) high or low note. These specified numbers, specific notes, etc. can be set by user operation, etc., or can be predetermined.

[0136] In the case of step S112 - 5 -Yes, the electronic musical instrument 10 determines that the lyrics are maintained (step S112 - 12 ). In the case of step S112 - 5 -No, the electronic musical instrument 10 determines that the lyrics are advanced (step S112 - 11 ).

[0137] According to the process of step S112 - 4 , when a plurality of keys are pressed with the intention of playing a chord, it is not preferable to advance the lyrics by the number of keys, and the lyrics can be advanced by only one key.

[0138] according to Figure 7 Such lyrics advancement determination processing can advance the lyrics if, for example, the lyrics are not multiple notes that are pronounced with a small time difference (so-called simultaneous chords (harmony)) but multiple notes that are pronounced with a large time difference (melody).

[0139] For example, when the highest note is pressed and the chord is pressed together (step S112-2-Yes), the lyrics can be advanced according to the highest note. In addition, as long as the highest note of the chord responsible for the melody is maintained, it is possible to control so that the lyrics are not advanced. This is expected to be effective when reproducing a performance of a polyphonic chorus.

[0140] Furthermore, when the lowest key is changed (step S112-5-Yes), control can be performed so that the lyrics are not advanced according to the lowest key. According to this structure, even if the pitch of only the lowest note of the chord of the bass part of the four-voice chorus changes, the lyrics are not advanced as long as the chord of the high part is maintained.

[0141] Furthermore, when a key other than the lowest note is changed (step S112-5-No), control can be performed to advance the lyrics according to the key. According to this configuration, when the part that can be responsible for the melody in a four-voice chorus is played independently instead of in chords, the lyrics can be appropriately advanced.

[0142] In addition, the question "whether the newly pressed note is higher than SKO" in step S112-2 may be replaced by "whether the newly pressed note corresponds to the melody part".

[0143] In addition, step S112-5 "whether the current number of keys is greater than the specified number and the sound of the new key is equivalent to a specific tone among the full tones of the keys" can also be replaced by "whether the sound of the new key is not equivalent to the melody part (or equivalent to the harmony part)".

[0144] It is also possible to provide information in advance about which note corresponds to the melody (or harmony) part according to a certain range of lyrics. For example, the information may indicate that the melody part of the lyrics corresponding to lyrics index = 0 to 10 is the highest note among the notes pressed, and the melody part of the lyrics corresponding to lyrics index = 11 to 20 is the lowest note among the notes pressed.

[0145] The information may include at least one of information on which highest note corresponds to the melody (or harmony) part, information on which range (for example, hiA to hiG#) corresponds to the melody (or harmony) part, and the like.

[0146] Based on the above information, the electronic musical instrument 10 may recognize the highest note (soprano part) as the melody in section A and the third highest note (tenor part) as the melody in the chorus, for example, and utilize it for lyrics control.

[0147] Figure 8 : is a diagram showing an example of lyrics advancement controlled by lyrics advancement determination processing. In this example, a case where a user presses a key as shown in the musical score is described. For example, the treble clef score may be pressed by the user's right hand, and the bass clef score may be pressed by the user's left hand. In addition, "Sle", "e", "ping", "heav", "en", and "ly" correspond to lyrics indexes 1-6, respectively.

[0148] In addition, the chord discrimination time is assumed to be shorter than an eighth note (for example, a 32nd note). In addition, it is assumed that the predetermined number of the above-mentioned step S102-17 is 4 and the specific note is the lowest note.

[0149] First, at timing t1, four keys are pressed. Figure 7In the lyrics advancement determination process, if step S112-2 is "yes", it is determined in step S112-11 that the lyrics are advanced. Then, in step S114, the electronic musical instrument 10 increments the lyrics index by 1, and generates and outputs the lyrics "Sle" using the four-tone synthesized sound.

[0150] Next, at timing t2, the user moves his left hand to the "Re (D)" key while continuing to press the key in his right hand. This Re sound is equivalent to the lowest sound among the sounds that the electronic musical instrument 10 should produce at t2. Figure 7 The lyrics advance determination process is performed, and the result of step S112-5 is "Yes", and it is determined in step S112-12 that the lyrics are not advanced. In addition, the electronic musical instrument 10 generates and outputs the Re sound using the vowel (e) of "Sle" that is already being pronounced while maintaining the lyrics index. The electronic musical instrument 10 continues to pronounce the other three sounds.

[0151] Similarly, the electronic musical instrument 10 outputs the lyrics "e" with the sound corresponding to the 4th key at t3, and maintains the lyrics and updates only the lowest note at t4. In addition, the electronic musical instrument 10 outputs the lyrics "ping" with the sound corresponding to the 4th key at t5, and maintains the lyrics and updates only the lowest note at t6.

[0152] exist Figure 8 In the example of t1-t6, the lyrics of the high triad are assigned one section per note, and the lyrics advance with each key press. On the other hand, the bass part is assigned one section per two notes (flower singing), which is judged as the lowest of the four tones, so there is a place where the lyrics do not advance with each key press.

[0153] <Synchronous Processing>

[0154] The synchronization process may also be a process of matching the position of the lyrics with the current playback position of the song data (accompaniment). According to this process, the position of the lyrics can be appropriately moved when the position of the lyrics exceeds the limit due to excessive keystrokes or the position of the lyrics does not advance as expected due to insufficient keystrokes.

[0155] Fig. 9 This is a diagram showing an example of a flowchart of synchronization processing.

[0156] The electronic musical instrument 10 obtains the playback position of the song data (step S126 - 1 ) and determines whether the playback position matches the (n+1)th singing voice playback position (step S126 - 2 ).

[0157] The n+1th singing voice reproduction position may indicate the timing at which the n+1th note is desired to be reproduced, which is derived in consideration of the total note length of the singing voice data up to the nth note, and the like.

[0158] If the playback position of the song data coincides with the n+1th singing voice playback position (step S126-2-yes), the synchronization process may be terminated. If not (step S126-2-no), the electronic musical instrument 10 may obtain the Xth singing voice playback position closest to the playback position of the song data (step S126-3), substitute X-1 in n (step S126-4), and terminate the synchronization process.

[0159] Furthermore, when the accompaniment is not reproduced, the synchronization process may be omitted. Furthermore, when the pronunciation timing of the appropriate lyrics is derived based on the singing voice data, even if the accompaniment is not reproduced, the electronic musical instrument 10 may perform a process of aligning the position of the lyrics appropriately to the position when the lyrics are pronounced, based on the elapsed time from the start of the performance to the present, the number of key presses, etc.

[0160] According to the embodiment described above, even when a plurality of keys are pressed simultaneously, the lyrics can be advanced satisfactorily.

[0161] (Variation Example)

[0162] exist Figure 4 , Figure 5 The sound synthesis processing shown in the above may be switched on / off based on the user's operation on the switch panel 140b. When it is OFF, the waveform data output unit 211 may be controlled to generate and output a sound source signal of musical instrument sound data of a pitch corresponding to the key.

[0163] exist Figure 6 In the flowcharts of the above, some steps may be omitted. When the determination process is omitted, the determination may be interpreted as proceeding along a route of always "yes" or always "no" in the flowchart.

[0164] The electronic musical instrument 10 only needs to be able to control the position of the lyrics, and may not necessarily generate or output the sound corresponding to the lyrics. For example, the electronic musical instrument 10 may send the sound waveform data generated based on the keystrokes to an external device (server computer 300, etc.), and the external device may generate / output a synthesized sound based on the sound waveform data.

[0165] The electronic musical instrument 10 may also control the display 150d to display lyrics. For example, lyrics near the current lyrics position (lyrics index) may be displayed, or lyrics corresponding to the sounds being pronounced or the sounds already pronounced may be displayed in color, etc., so that the current lyrics position can be identified.

[0166] The electronic musical instrument 10 may transmit at least one of the singing voice data and information about the current position of lyrics to the external device. The external device may control its own display to display the lyrics based on the received singing voice data and information about the current position of lyrics.

[0167] In the above example, the electronic musical instrument 10 is a keyboard instrument such as a keyboard, but the present invention is not limited thereto. The electronic musical instrument 10 may be an electric violin, electric guitar, drum, trumpet, etc., as long as it has a structure that allows the user to specify the timing of sound generation.

[0168] Therefore, the "key" of the present invention can also be replaced by a string, a valve, other performance operating parts for specifying pitch, any performance operating parts, etc. The "key pressing" of the present invention can also be replaced by keying, picking, playing, operating parts, etc. The "key releasing" of the present invention can also be replaced by stopping the string, stopping the playing, stopping the operating parts (non-operation), etc.

[0169] In addition, the block diagram used in the description of the above-mentioned embodiment represents a block of a functional unit. These functional blocks (constituents) are implemented by any combination of hardware and / or software. In addition, the implementation means of each functional block is not particularly limited. That is, each functional block can be implemented by a physically combined device, or two or more physically separated devices can be connected by wire or wirelessly and implemented by these multiple devices.

[0170] Furthermore, terms used to explain the present invention and / or terms necessary for understanding the present invention may be replaced with terms having the same or similar meanings.

[0171] The information, parameters, etc. described in the present invention may be expressed using absolute values, relative values ​​relative to a specified value, or other corresponding information. In addition, the names used in the parameters, etc. in the present invention are not restrictive in any way.

[0172] The information, signals, etc. described in the present invention may also be represented by any of a variety of different technologies. For example, data, commands, instructions, information, signals, bits, symbols, chips, etc. mentioned throughout the above description may also be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, light fields or photons, or any combination thereof.

[0173] Information, signals, etc. may also be input and output via multiple network nodes. The input and output information, signals, etc. may be stored in a specific location (such as a memory) or managed using a table. The input and output information, signals, etc. may be overwritten, updated, or appended. The output information, signals, etc. may also be deleted. The input information, signals, etc. may also be sent to other devices.

[0174] Whether software is referred to as software, firmware, middleware, microcode, hardware description language, or by other names, it should be broadly interpreted as referring to commands, instruction sets, codes, code segments, program codes, programs, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, executables, execution threads, sequences, functions, etc.

[0175] In addition, software, commands, information, etc. may also be sent and received via a transmission medium. For example, when the software is sent from a website, server, or other remote source using at least one of wired technology (coaxial cable, optical cable, twisted pair, digital subscriber line (DSL), etc.) and wireless technology (infrared, microwave, etc.), at least one of these wired technologies and wireless technologies is included in the definition of transmission medium.

[0176] The various technical solutions / implementations described in the present invention may be used alone, in combination, or switched as they are executed. In addition, the processing sequence, order, flow chart, etc. of the various technical solutions / implementations described in the present invention may be replaced in order as long as there is no contradiction. For example, with respect to the method described in the present invention, various step elements are shown using an illustrative order, and are not limited to the specific order shown.

[0177] The term “based on” used in the present invention does not mean “based only on” unless otherwise specified. In other words, the term “based on” means both “based only on” and “based at least on”.

[0178] Any reference to an element using the terms "first", "second", etc. used in the present invention does not limit the quantity or order of these elements as a whole. These terms can be used in the present invention as a convenient method to distinguish between two or more elements. Therefore, the reference to the first and second elements does not mean that only two elements can be used, or that the first element precedes the second element in some form.

[0179] In the present invention, when the terms "include", "including" and their variations are used, these terms are inclusive, like the term "comprising". Furthermore, the term "or" used in the present invention does not mean an exclusive logical and.

[0180] In the present invention, for example, when an article is added by translation, the present invention may also include a case where the noun following the article is in plural form.

[0181] The invention related to the present invention has been described in detail above, but it is obvious to those skilled in the art that the invention related to the present invention is not limited to the embodiments described in the present invention. The invention related to the present invention can be implemented as a modified and altered form as long as it does not depart from the gist and scope of the invention determined based on the claims. Therefore, the description of the present invention is for the purpose of illustrative description and does not bring any limiting meaning to the invention related to the present invention.

Claims

1. An electronic musical instrument, characterized in that: have: A plurality of performance operating elements are respectively associated with different pitch data; and at least one processor; The at least one processor performs the following processing: According to the user operation, determining whether two or more performance operating elements are operated within a set time; When it is determined that two or more musical performance operating elements are operated within a set time, instructing to pronounce a singing voice corresponding to the same lyrics at each pitch corresponding to the two or more musical performance operating elements specified by the user operation; In the above process, When it is determined that two or more musical performance operating elements have been operated within the set time, determining whether the pitch corresponding to the newly operated musical performance operating element among the two or more musical performance operating elements is the lowest pitch among the plurality of high pitches specified by the user operation; In the case where it is judged to be the lowest tone, it is judged that the lyrics are maintained; When it is determined that the tone is other than the lowest tone, it is determined that the lyrics are moving forward.

2. The electronic musical instrument according to claim 1, wherein: The at least one processor instructs the vocalization of a singing voice corresponding to the corresponding lyrics at a pitch designated by the user operation when it is determined that two or more performance operating elements have not been operated within a set time.

3. The electronic musical instrument according to claim 1 or 2, characterized in that: The at least one processor determines whether the lyrics are advanced or maintained based on a user operation on any one of the plurality of performance operating elements.

4. The electronic musical instrument according to claim 1 or 2, characterized in that: The at least one processor performs the following processing: Instructing reproduction of accompaniment data; According to the user operation, determining whether all pitch designations are released; When it is determined that all pitch designations are released, the first reproduction position of the singing text data corresponding to the lyrics to be sung by the next user operation is changed to a second reproduction position corresponding to the reproduction position in the accompaniment data.

5. The electronic musical instrument according to claim 1 or 2, characterized in that: The at least one processor performs the following processing: According to user operation, obtain instrument sound data; The above-mentioned instrument sound data is provided with formant information corresponding to the lyrics.

6. The electronic musical instrument according to claim 5, characterized in that: The at least one processor adds formant information output by the trained model when lyrics data is input to the trained model to the musical instrument sound data.

7. The electronic musical instrument according to claim 6, wherein: The trained model is generated by performing machine learning using a singer's voice data as training data, and outputs the resonance peak information representing the acoustic feature quantity of the singer's voice based on the input of the lyrics data.

8. A method in an electronic musical instrument, characterized in that, At least one processor of the electronic musical instrument performs the following processing: According to the user operation, determining whether two or more performance operating elements are operated within a set time; When it is determined that two or more musical performance operating elements are operated within a set time, instructing to pronounce a singing voice corresponding to the same lyrics at each pitch corresponding to the two or more musical performance operating elements specified by the user operation; In the above process, When it is determined that two or more musical performance operating elements have been operated within the set time, determining whether the pitch corresponding to the newly operated musical performance operating element among the two or more musical performance operating elements is the lowest pitch among the plurality of high pitches specified by the user operation; In the case where it is judged to be the lowest tone, it is judged that the lyrics are maintained; When it is determined that the tone is other than the lowest tone, it is determined that the lyrics are moving forward.

9. The method according to claim 8, characterized in that The at least one processor instructs the vocalization of a singing voice corresponding to the corresponding lyrics at a pitch designated by the user operation when it is determined that two or more performance operating elements have not been operated within a set time.

10. The method according to claim 8 or 9, characterized in that The at least one processor determines whether the lyrics are advanced or maintained based on a user operation on any one of the plurality of performance operating elements.

11. The method according to claim 8 or 9, characterized in that: The at least one processor performs the following processing: Instructing reproduction of accompaniment data; According to the user operation, determining whether all pitch designations are released; When it is determined that the designation of all pitches has been released, the first reproduction position of the lyrics to be sung in response to the next user operation is changed to a second reproduction position corresponding to the reproduction position in the accompaniment data.

12. The method according to claim 8 or 9, characterized in that The at least one processor performs the following processing: According to user operation, obtain instrument sound data; The above-mentioned instrument sound data is provided with formant information corresponding to the lyrics.

13. The method according to claim 12, characterized in that The at least one processor adds formant information output by the trained model when lyrics data is input to the trained model to the musical instrument sound data.

14. The method according to claim 13, characterized in that The trained model is generated by performing machine learning using a singer's voice data as training data, and outputs the resonance peak information representing the acoustic feature quantity of the singer's voice based on the input of the lyrics data.

15. A storage medium, characterized in that: A program is stored in which at least one processor of an electronic musical instrument executes the following processing: According to the user operation, determining whether two or more performance operating elements are operated within a set time; When it is determined that two or more musical performance operating elements are operated within a set time, instructing to pronounce a singing voice corresponding to the same lyrics at each pitch corresponding to the two or more musical performance operating elements specified by the user operation; In the above process, When it is determined that two or more musical performance operating elements have been operated within the set time, determining whether the pitch corresponding to the newly operated musical performance operating element among the two or more musical performance operating elements is the lowest pitch among the plurality of high pitches specified by the user operation; In the case where it is judged to be the lowest tone, it is judged that the lyrics are maintained; When it is determined that the tone is other than the lowest tone, it is determined that the lyrics are moving forward.

Images