Electronic musical instrument and control method of electronic musical instrument

Abstract

An electronic musical instrument and a method for controlling the same. The electronic musical instrument comprises a performance operating member (140k) and a processor (306), wherein the processor (306) performs control in the following manner: regardless of whether the user operation on the performance operating member (140k) is detected at a timing when the user operation is to be detected, singing synthesis data (217) is generated according to lyrics data corresponding to the timing; when the user operation is detected at the timing (S106: Yes), the singing according to the generated singing synthesis data (217) is allowed to be emitted (S109); and when the user operation is not detected at the timing (S106: No), the singing according to the generated singing synthesis data (217) is not allowed to be emitted (S115).

Description

Technical Field

[0001] The present disclosure relates to an electronic musical instrument and a control method of the electronic musical instrument. Background Art

[0002] A technique for advancing lyrics in synchronization with a performance based on user operations using a keyboard or the like is disclosed (for example, Japanese Patent No. 4735544). Summary of the invention

[0003] An electronic musical instrument as an embodiment of the present invention includes a performance operating member and a processor, and the processor is controlled in the following manner: regardless of whether the user operation on the performance operating member is detected at the timing when the user operation should be detected, singing synthesis data is generated according to lyrics data corresponding to the timing; when the user operation is detected at the timing, singing according to the generated singing synthesis data is allowed to be emitted, and when the user operation is not detected at the timing, singing according to the generated singing synthesis data is not allowed to be emitted.

[0004] As a control method for an electronic musical instrument of one embodiment of the present invention, at least one processor of the electronic musical instrument is controlled in the following manner: regardless of whether the user operation is detected at the timing when the user operation should be detected, singing synthesis data is generated according to the lyrics data corresponding to the timing; when the user operation is detected at the timing, singing according to the generated singing synthesis data is allowed to be emitted; when the user operation is not detected at the timing, singing according to the generated singing synthesis data is not allowed to be emitted.

[0005] An electronic musical instrument as an embodiment of the present invention comprises: a performance operating member; and at least one processor, wherein the at least one processor instructs the emission of a song corresponding to first character data based on the detection of a user operation corresponding to a first timing, wherein the first character data is the first character data in lyrics data including the first character data corresponding to the first timing, the second character data corresponding to a second timing after the first timing, and the third character data corresponding to a third timing after the second timing, and when the user operation corresponding to the third timing is detected but the user operation corresponding to the second timing is not detected, the at least one processor instructs the emission of a song corresponding to the third character data instead.

[0006] As an embodiment of the present invention, a control method for an electronic musical instrument, wherein at least one processor of the electronic musical instrument instructs the emission of a song corresponding to first character data based on the detection of a user operation corresponding to a first timing, wherein the first character data is the first character data in lyrics data including the first character data corresponding to the first timing, the second character data corresponding to a second timing after the first timing, and the third character data corresponding to a third timing after the second timing, and when the user operation corresponding to the third timing is detected while the user operation corresponding to the second timing is not detected, the emission of a song corresponding to the second character data is not instructed, but the emission of a song corresponding to the third character data is instructed.

[0007] According to one embodiment of the present invention, the progression of lyrics involved in performance can be appropriately controlled. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 FIG. 1 is a diagram showing an example of the appearance of an electronic musical instrument 10 according to an embodiment.

[0009] Figure 2 This is a diagram showing an example of the hardware configuration of a control system 200 of an electronic musical instrument 10 according to an embodiment.

[0010] Figure 3 1 is a diagram showing a configuration example of the sound learning unit 301 according to one embodiment.

[0011] Figure 4 It is a diagram showing an example of the waveform data output unit 211 according to one embodiment.

[0012] Figure 5 It is a diagram showing another example of the waveform data output unit 211 according to one embodiment.

[0013] Figure 6 This is a diagram showing an example of a flowchart of a lyrics progression control method according to one embodiment.

[0014] Figure 7 This is a diagram showing an example of the progression of lyrics controlled using the lyrics progression control method according to one embodiment. DETAILED DESCRIPTION

[0015] The present inventors conceived of generating singing voice waveform data independently of a user's performance operation and controlling permission and non-permission of emitting sound corresponding to the singing voice waveform data, and came up with the electronic musical instrument of the present disclosure.

[0016] According to one aspect of the present disclosure, the progression of spoken lyrics can be easily controlled based on a user's operation.

[0017] Hereinafter, the embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the following description, the same reference numerals are used for the same parts. The names and functions of the same parts are the same, so the detailed description will not be repeated.

[0018] (Electronic Musical Instruments)

[0019] Figure 1 140 is a diagram showing an example of the appearance of the electronic musical instrument 10 according to one embodiment. The electronic musical instrument 10 may be equipped with a switch (button) panel 140b, a keyboard 140k, a pedal 140p, a display 150d, a speaker 150s, and the like.

[0020] The electronic musical instrument 10 is a device for receiving input from the user via operating elements such as a keyboard and a switch to control the performance, the progression of lyrics, etc. The electronic musical instrument 10 may also be a device having the function of generating sounds corresponding to performance information such as MIDI (Musical Instrument Digital Interface) data. The device may be an electronic musical instrument (electronic piano, synthesizer, etc.) or an analog musical instrument having the functions of the above-mentioned operating elements equipped with sensors, etc.

[0021] The switch panel 140b may include switches for specifying volume, setting sound source, timbre, etc., selecting songs (accompaniment), starting / stopping song playback, setting song playback (rhythm, etc.), etc.

[0022] The keyboard 140k may also have a plurality of keys as performance operating elements. The pedal 140p may be a delay pedal having a function of prolonging the sound of a pressed keyboard while the pedal is depressed, or may be a pedal for operating an effector for processing the tone, volume, and the like.

[0023] Furthermore, in the present disclosure, the delay pedal, pedal, foot switch, controller (operating member), switch, button, touch panel, etc. may be replaced with each other. The stepping of the pedal in the present disclosure may also be replaced by the operation of the controller.

[0024] The keys may also be referred to as playing operators, pitch operators, tone operators, direct operators, first operators, etc. The pedals may also be referred to as non-playing operators, non-pitch operators, non-tone operators, indirect operators, second operators, etc.

[0025] The display 150d may also display lyrics, music scores, various setting information, etc. The speaker 150s may also be used to emit sounds generated by the performance.

[0026] Furthermore, the electronic musical instrument 10 may generate or convert at least one of a MIDI message (event) and an Open Sound Control (OSC) message.

[0027] The electronic musical instrument 10 may also be referred to as a control device 10 , a lyrics progression control device 10 , or the like.

[0028] The electronic musical instrument 10 may communicate with a network (such as the Internet) via at least one of wired and wireless (for example, Long Term Evolution (LTE), 5th generation mobile communication system New Radio (5G NR), Wi-Fi (registered trademark), etc.).

[0029] The electronic musical instrument 10 may store singing data (also referred to as lyrics text data, lyrics information, etc.) related to the lyrics that are the control object of the movement in advance, or may send and / or receive via a network. The singing data may be text recorded in a musical notation language (e.g., Music XML), may be expressed in a MIDI data storage format (e.g., Standard MIDI File (SMF) format), or may be text provided by a common text file. The singing data may also be singing data 215 described later. In the present disclosure, singing, speech, sound, etc. may also be interchangeable.

[0030] Alternatively, the electronic musical instrument 10 may acquire content of a user's singing in real time via a microphone or the like included in the electronic musical instrument 10 , and acquire text data obtained by applying voice recognition processing thereto as singing voice data.

[0031] Figure 2 This is a diagram showing an example of the hardware configuration of a control system 200 of an electronic musical instrument 10 according to an embodiment.

[0032] Central Processing Unit (CPU) 201, ROM (Read Only Memory) 202, RAM (Random Access Memory) 203, waveform data output unit 211, Figure 1 The switch (button) panel 140b, the keyboard 140k, the key scanner 206 connected to the pedal 140p, and the Figure 1 An LCD controller 208 connected to an LCD (Liquid Crystal Display) which is an example of the display 150 d is connected to the system bus 209 .

[0033] The CPU 201 may be connected to a timer 210 (also referred to as a counter) for controlling the performance. The timer 210 may also be used to count the progress of the automatic performance in the electronic musical instrument 10. The CPU 201 may be referred to as a processor, and may also include an interface with peripheral circuits, a control circuit, an operation circuit, and a register, etc.

[0034] The functions in each device can also be realized by reading the specified software (program) into hardware such as the processor 1001 and the memory 1002, so that the processor 1001 performs calculations to control the communication of the communication device 1004, the reading and / or writing of data in the memory 1002 and the storage 1003, etc.

[0035] The CPU 201 uses the RAM 203 as a working memory and executes the control program stored in the ROM 202 to perform Figure 1 The control operation of the electronic musical instrument 10. In addition, the ROM 202 can store vocal data, accompaniment data, and music (song) data including them, in addition to the above-mentioned control program and various fixed data.

[0036] The waveform data output unit 211 may also include a sound source LSI (large scale integrated circuit) 204, a sound synthesis LSI 205, etc. The sound source LSI 204 and the sound synthesis LSI 205 may also be combined into one LSI. Figure 3 In addition, part of the processing of the waveform data output unit 211 may be performed by the CPU 201 or may be performed by the CPU included in the waveform data output unit 211 .

[0037] The singing waveform data 217 and the song waveform data 218 output from the waveform data output unit 211 are converted into analog singing sound output signals and analog musical sound output signals by D / A converters 212 and 213, respectively. The analog musical sound output signal and the analog singing sound output signal can also be mixed in the mixer 214, and after the mixed signal is amplified by the amplifier 215, it is output from the speaker 150s or the output terminal. In addition, the singing waveform data can also be called singing synthesis data. Although not shown in the figure, the singing waveform data 217 and the song waveform data 218 can also be converted into analog by a D / A converter after being digitally synthesized to obtain a mixed signal.

[0038] The key scanner (scanner) 206 scans stably Figure 1 The key-press / key-release state of the keyboard 140k, the switch operation state of the switch panel 140b, the pedal operation state of the pedal 140p, etc., are interrupted to the CPU 201 to transmit the state change.

[0039] The LCD controller 208 is an IC (Integrated Circuit) that controls the display state of the LCD as an example of the display 150 d .

[0040] In addition, the system structure is an example and is not limited to this. For example, the number of circuits included is not limited to this. The electronic musical instrument 10 may have a structure that does not include a part of the circuit (mechanism), or may have a structure that realizes the function of one circuit by multiple circuits. It may also have a structure that realizes the functions of multiple circuits by one circuit.

[0041] In addition, the electronic musical instrument 10 may be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array), and a part or all of each functional block may be implemented by the hardware. For example, the CPU 201 may be implemented by at least one of these hardware.

[0042] <Creation of acoustic model>

[0043] Figure 3 1 is a diagram showing an example of the structure of the voice learning unit 301 according to one embodiment. The voice learning unit 301 may also be installed as Figure 1 The sound learning unit 301 is a function executed by the server computer 300 which exists outside the electronic musical instrument 10 separately. In addition, the sound learning unit 301 may be built into the electronic musical instrument 10 as a function executed by the CPU 201, the sound synthesis LSI 205, etc.

[0044] The sound learning section 301 and the waveform data output section 211 that realize the sound synthesis in the present disclosure can be respectively realized based on, for example, a statistical sound synthesis technology based on deep learning.

[0045] The speech learning unit 301 may include a learning text analysis unit 303 , a learning acoustic feature extraction unit 304 , and a model learning unit 305 .

[0046] The voice learning unit 301 uses, for example, data obtained by recording a singer singing a plurality of songs of appropriate genres as learning singing voice data 312. In addition, lyrics texts of the songs are prepared as learning singing voice data 311.

[0047] The learning text analysis unit 303 inputs the learning singing data 311 including the lyrics text and analyzes the data. As a result, the learning text analysis unit 303 estimates and outputs a learning language feature value sequence 313 which is a discrete numerical sequence representing the phonemes, pitches, etc. corresponding to the learning singing data 311.

[0048] The learning sound feature extraction unit 304 inputs and analyzes the learning singing voice data 312, which is sound data recorded through a microphone or the like by a singer singing a lyrics text corresponding to the learning singing voice data 311 based on the input of the learning singing voice data 311. As a result, the learning sound feature extraction unit 304 extracts and outputs a learning sound feature sequence 314 representing the features of the sound corresponding to the learning singing voice data 312.

[0049] In the present disclosure, the acoustic feature sequence corresponding to the learning acoustic feature sequence 314 and the acoustic feature sequence 317 described later includes acoustic feature data (also referred to as formant information, spectrum information, etc.) obtained by modeling the vocal tract of a person, and vocal cord sound source data (also referred to as sound source information) obtained by modeling the vocal cords of a person. As the spectrum information, for example, Mel cepstrum, line spectrum pairs (LSP), etc. can be used. As the sound source information, the fundamental frequency (F0) and power value representing the pitch frequency of the human voice can be used.

[0050] The model learning unit 305 estimates the acoustic model that maximizes the probability of generating the learning acoustic feature sequence 314 from the learning language feature sequence 313 by machine learning. That is, the relationship between the language feature sequence as text and the acoustic feature sequence as sound is expressed by a statistical model such as an acoustic model. The model learning unit 305 outputs the model parameters for expressing the acoustic model calculated as a result of the machine learning as a learning result 315. Therefore, the acoustic model corresponds to the trained model.

[0051] As the acoustic model represented by the learning result 315 (model parameter), HMM (Hidden Markov Model) may be used.

[0052] When a singer sings lyrics to a certain melody, the HMM acoustic model can also be used to learn how the characteristic parameters of the vocal cord vibration or vocal tract characteristics change over time while the vocalization is being performed. More specifically, the HMM acoustic model can also be a model that models the spectrum, fundamental frequency, and their time structure obtained from the singing voice data for learning in phoneme units.

[0053] First, the HMM acoustic model is used to describe Figure 3 The model learning unit 305 in the sound learning unit 301 may also learn the HMM sound model with the maximum likelihood by inputting the learning language feature sequence 313 output by the learning text analysis unit 303 and the learning sound feature sequence 314 output by the learning sound feature extraction unit 304.

[0054] The spectral parameters of the singing voice can be modeled by continuous HMM. On the other hand, the logarithmic fundamental frequency (F0) is a time series signal of variable dimension that takes continuous values ​​in the voiced interval and has no value in the silent interval, so it cannot be directly modeled by the usual continuous HMM or discrete HMM. Therefore, MSD-HMM (Multi-Space probability Distribution HMM), which is an HMM based on probability distribution on multiple spaces corresponding to variable dimensions, is used as the spectral parameter, and the Mel-cepstrum is used as a multi-dimensional Gaussian distribution, the voiced sound (with sound) of the logarithmic fundamental frequency (F0) is used as a 1-dimensional space, and the unvoiced sound (without sound) is modeled as a Gaussian distribution of a 0-dimensional space at the same time.

[0055] In addition, it is known that the characteristics of the phonemes that constitute singing voices vary due to various factors even if the phonemes are the same. For example, the spectrum and logarithmic fundamental frequency (F0) of the phoneme, which is the basic phonological unit, vary depending on the singing style and rhythm, or the preceding and following lyrics and pitch, etc. The content of the main factor that affects such acoustic feature quantities is called context.

[0056] In the statistical sound synthesis processing of one embodiment, in order to model the acoustic features of the sound with high accuracy, an HMM acoustic model (context-dependent model) that takes context into consideration may also be used. Specifically, the learning text analysis unit 303 may output not only the phonemes and pitches of each frame, but also a learning language feature quantity sequence 313 that takes into consideration the immediately preceding and immediately succeeding phonemes, the current position, the immediately preceding and immediately succeeding vibrato, the tone, etc. In addition, in order to improve the efficiency of context combination, context clustering based on a decision tree may be used.

[0057] For example, the model learning unit 305 may also generate a state duration length decision tree for determining the state duration length as a learning result 315 based on the learning language feature value sequence 313, wherein the learning language feature value sequence 313 corresponds to the context of a large number of phonemes related to the state duration length extracted by the learning text analysis unit 303 from the learning singing data 311.

[0058] In addition, for example, the model learning unit 305 can also generate a mel-reciprocal parameter decision tree for determining the mel-reciprocal parameter as a learning result 315 based on the learning sound feature sequence 314, wherein the learning sound feature sequence 314 corresponds to a large number of phonemes related to the mel-reciprocal parameters extracted by the learning sound feature extraction unit 304 from the learning singing sound data 312.

[0059] Furthermore, for example, the model learning unit 305 may generate a logarithmic fundamental frequency decision tree for determining the logarithmic fundamental frequency (F0) as the learning result 315 based on the learning acoustic feature quantity sequence 314 corresponding to a large number of phonemes related to the logarithmic fundamental frequency (F0) extracted from the learning singing voice data 312 by the learning acoustic feature quantity extraction unit 304. Furthermore, the voiced interval and the unvoiced interval of the logarithmic fundamental frequency (F0) may be modeled as 1-dimensional and 0-dimensional Gaussian distributions, respectively, by MSD-HMM corresponding to the variable dimension, and the logarithmic fundamental frequency decision tree may be generated.

[0060] In addition, an acoustic model based on a deep neural network (DNN) may be used instead of an acoustic model based on HMM, or may be used together with an acoustic model based on HMM. In this case, the model learning unit 305 may generate a model parameter representing a nonlinear conversion function of each neuron in the DNN from the language feature quantity to the acoustic feature quantity as the learning result 315. According to the DNN, it is possible to express the relationship between the language feature quantity sequence and the acoustic feature quantity sequence using a complex nonlinear conversion function that is difficult to express with a decision tree.

[0061] The acoustic model of the present disclosure is not limited to these, and any sound synthesis method may be adopted as long as it is a technique using statistical sound synthesis processing, such as an acoustic model combining HMM and DNN.

[0062] For example, Figure 3 As shown, in Figure 1 When the electronic musical instrument 10 is shipped from the factory, the learning result 315 (model parameter) can also be stored in Figure 2 When the power of the electronic musical instrument 10 is turned on, the ROM 202 of the control system of the electronic musical instrument 10 is Figure 2 The ROM 202 is loaded into the singing voice control unit 307 described later in the waveform data output unit 211, etc.

[0063] For example, Figure 3 As shown, the learning result 315 can also be downloaded from the outside such as the Internet to the singing voice control unit 307 in the waveform data output unit 211 through the network interface 219 by the performer operating the switch panel 140b of the electronic musical instrument 10.

[0064] <Sound synthesis based on acoustic model>

[0065] Figure 4 It is a diagram showing an example of the waveform data output unit 211 according to one embodiment.

[0066] The waveform data output unit 211 includes a processing unit (also called a text processing unit, a pre-processing unit, etc.) 306, a singing control unit (also called an acoustic model unit) 307, a sound source 308, a singing synthesis unit (also called a vocalization model unit) 309, and a mute unit 310, etc.

[0067] The waveform data output unit 211 is based on Figure 1 The keyboard has 140k keys, and input is via Figure 2 The key scanner 206 of the singing voice control unit 307 synthesizes and outputs the singing voice waveform data 217 corresponding to the lyrics and pitch from the singing voice data 215 including the information of the lyrics and the pitch indicated by the CPU 201. In other words, the waveform data output unit 211 performs a statistical sound synthesis process that predicts the singing voice waveform data 217 corresponding to the singing voice data 215 including the lyrics text by using a statistical model such as the acoustic model set in the singing voice control unit 307.

[0068] In addition, when the song data is reproduced, the waveform data output unit 211 outputs the song waveform data 218 corresponding to the corresponding song reproduction position. Here, the song data may be equivalent to the data of the accompaniment (for example, data on the pitch, timbre, sound timing, etc. of one or more sounds), the data of the accompaniment and melody, and may also be called back track data, etc.

[0069] The processing unit 306 inputs, for example, the result of the performance of the performer in coordination with the automatic performance, the result of the performance of the performer including the result of the performance of the performer Figure 2 The CPU 201 may also analyze the singing voice data 215 of information related to the phonemes and pitches of the lyrics specified by the CPU 201. The singing voice data 215 may also include data (such as pitch data and note length data) of the nth note (also referred to as the nth note, the nth timing, etc.), data of the nth lyrics corresponding to the nth note, etc.

[0070] For example, the processing unit 306 may determine whether or not there is a lyrics progression based on the note on / off data and pedal on / off data obtained from the operation of the keyboard 140k and the pedal 140p, based on the lyrics progression control method described later, and obtain the singing voice data 215 corresponding to the lyrics to be output. In addition, the processing unit 306 may analyze the language feature quantity sequence 316 representing phonemes, word classes, words, etc. corresponding to the pitch data specified by the key or the pitch data of the acquired singing voice data 215, and the character data of the acquired singing voice data 215, and output it to the singing voice control unit 307.

[0071] The singing data may include at least one of the following information: lyrics (characters), syllable type (starting syllable, middle syllable, ending syllable, etc.), lyrics index, corresponding pitch (correct pitch), and corresponding pronunciation period (e.g., pronunciation start timing, pronunciation end timing, and pronunciation length (duration)).

[0072] For example, in Figure 4 In the example of , the singing voice data 215 may also include lyrics data of the nth lyrics corresponding to the nth (n=1, 2, 3, 4, ...) note, and information on the specified timing (nth lyrics reproduction position) at which the nth note should be reproduced. The singing voice data of the nth lyrics may also be referred to as the nth lyrics data. The nth lyrics data may also include data of characters included in the nth lyrics (character data of the nth lyrics data), pitch data corresponding to the nth lyrics (pitch data of the nth lyrics data), and information such as the length of the sound corresponding to the nth lyrics.

[0073] The vocal data 215 may also include information (data in a specific sound file format, MIDI data, etc.) for playing the accompaniment (song data) corresponding to the lyrics. When the vocal data is represented in the SMF format, the vocal data 215 may also include a track block storing data related to the vocals and a track block storing data related to the accompaniment. The vocal data 215 may also be read from the ROM 202 to the RAM 203. The vocal data 215 is stored in a memory (e.g., the ROM 202, the RAM 203) before the performance.

[0074] In addition, the electronic musical instrument 10 can also control the progression of automatic accompaniment based on events represented by the singing data 215 (for example, meta-events (timing information) indicating the vocalization timing and pitch of lyrics, MIDI events indicating notes on or off, or meta-events indicating beats, etc.).

[0075] The singing voice control unit 307 estimates the corresponding sound feature sequence 317 based on the language feature sequence 316 input from the processing unit 306 and the sound model set as the learning result 315, and outputs the resonance peak information 318 corresponding to the estimated sound feature sequence 317 to the singing voice synthesis unit 309.

[0076] For example, when the HMM sound model is adopted, the singing voice control unit 307 connects the HMM according to each context reference decision tree obtained from the language feature sequence 316, and predicts the sound feature sequence 317 (resonance peak information 318 and vocal sound source data 319) with the highest output probability based on the connected HMMs.

[0077] When the DNN acoustic model is used, the singing voice control unit 307 may output the acoustic feature value sequence 317 in units of frames for the phoneme string of the language feature value sequence 316 input in units of frames.

[0078] exist Figure 4 In the process, the processing unit 306 obtains musical instrument sound data (pitch information) corresponding to the pitch of the pressed sound from the memory (which may be either the ROM 202 or the RAM 203 ), and outputs it to the sound source 308 .

[0079] The sound source 308 generates a sound source signal (also referred to as musical instrument sound waveform data) of musical instrument sound data (pitch information) corresponding to the sound to be emitted (note-on) based on the note-on / off data input from the processing unit 306, and outputs it to the singing voice synthesis unit 309. The sound source 308 may also perform control processing such as envelope control of the emitted sound.

[0080] The singing voice synthesis unit 309 forms a digital filter for vocal tract modeling based on the sequence of formant information 318 sequentially input from the singing voice control unit 307. In addition, the singing voice synthesis unit 309 uses the sound source signal input from the sound source 308 as an excitation source signal, applies the digital filter, and generates and outputs the singing voice waveform data 217 of the digital signal. In this case, the singing voice synthesis unit 309 can be called a synthesis filter unit.

[0081] Furthermore, the singing voice synthesis unit 309 may adopt various voice synthesis methods including a cepstrum voice synthesis method and an LSP voice synthesis method.

[0082] The mute unit 310 may apply mute processing to the singing voice waveform data 217 output from the singing voice synthesis unit 309. For example, the mute unit 310 may not apply the mute processing when a note-on signal is input (i.e., a key is pressed), and may apply the mute processing when no note-on signal is input (i.e., all keys are off). The mute processing may also be processing to make the volume of the waveform 0 or muted (very small).

[0083] exist Figure 4 In the example, since the output singing waveform data 217 uses the instrument sound as the sound source signal, the fidelity is slightly lost compared to the singer's singing voice, but the singing voice becomes a voice in which both the atmosphere of the instrument sound and the sound quality of the singer's singing voice remain well, and effective singing waveform data 217 can be output.

[0084] In addition, the sound source 308 may operate in such a manner that it outputs the output of other channels together with the processing of the musical instrument sound waveform data as the song waveform data 218. Thus, the accompaniment sound may be produced as a normal musical instrument sound, or the musical instrument sound of the melody line may be produced while the singing voice of the melody is produced.

[0085] Figure 5 2 is a diagram showing another example of the waveform data output unit 211 according to one embodiment. Figure 4 Repeated content will not be described again.

[0086] As mentioned above, Figure 5 The singing voice control unit 307 estimates the acoustic feature sequence 317 based on the acoustic model. Then, the singing voice control unit 307 outputs the formant information 318 corresponding to the estimated acoustic feature sequence 317 and the vocal cord sound source data (pitch information) 319 corresponding to the estimated acoustic feature sequence 317 to the singing voice synthesis unit 309. The singing voice control unit 307 may estimate the estimated value of the acoustic feature sequence 317 that maximizes the probability of generating the acoustic feature sequence 317.

[0087] The singing voice synthesis unit 309 can also generate data for producing the following signal (for example, it can also be called singing waveform data of the nth lyric corresponding to the nth note) and output it to the sound source 308. The signal is a signal obtained by applying a digital filter to the basic frequency (F0) and a pulse train that periodically repeats with a power value contained in the vocal cord sound source data 319 input from the singing voice control unit 307 (in the case of voiced phonemes), or white noise with a power value contained in the vocal cord sound source data 319 (in the case of unvoiced phonemes), or a mixture of them. The digital filter is obtained by modeling the vocal tract based on the sequence of resonance peak information 318.

[0088] like Figure 4 As shown, the mute unit 310 may apply mute processing to the singing voice waveform data 217 output from the singing voice synthesis unit 309 .

[0089] The sound source 308 generates and outputs the singing voice waveform data 217 as a digital signal from the singing voice waveform data of the n-th lyrics corresponding to the sound to be produced (note-on), based on the note-on / off data input from the processing unit 306 .

[0090] exist Figure 5 In the example, since the output singing waveform data 217 uses the sound generated by the sound source 308 as the sound source signal based on the vocal cord sound source data 319, it is a signal completely modeled by the singing control unit 307, which can make the singer's singing voice very fidelity and output the natural singing waveform data 217.

[0091] in addition, Figure 4 and Figure 5 The mute section 310 is located at a position where the output from the vocal synthesis section 309 is input, but the position of the mute section 310 is not limited thereto. For example, the mute section 310 may be configured at the output of the sound source 308 (or included in the sound source 308), and may mute the musical instrument sound waveform data or the vocal waveform data output from the sound source 308.

[0092] Thus, the sound synthesis disclosed in the present invention is different from the existing vocoder (a method of inputting human-spoken words through a microphone, replacing them with musical instrument sounds and synthesizing them), and even if the user (performer) does not sing in reality (in other words, even if the user does not input a real-time sound signal to the electronic musical instrument 10), the synthesized sound can be output through keyboard operation.

[0093] As described above, by adopting the technology of statistical sound synthesis processing as the sound synthesis method, it is possible to achieve a significantly smaller memory capacity than the conventional segment synthesis method. For example, in an electronic musical instrument of the segment synthesis method, a memory with a storage capacity of hundreds of megabytes is required for the sound segment data, but in the present embodiment, a memory with a storage capacity of only a few megabytes is required to store the model parameters of the learning result 315. Therefore, a lower-priced electronic musical instrument can be achieved, and a high-quality singing performance system can be utilized for a wider range of user groups.

[0094] Furthermore, in the conventional segment data method, since the segment data needs to be adjusted manually, the production of data for singing performance requires a huge amount of time (in years) and labor, but in the production of model parameters for the learning result 315 of the HMM acoustic model or the DNN acoustic model in this embodiment, data adjustment is almost unnecessary, so a fraction of the production time and labor is sufficient. Thus, a lower-priced electronic musical instrument can be realized.

[0095] In addition, general users can also use the learning function built into the available server computer 300, voice synthesis LSI 205, etc. as a cloud service to learn their own voice, the voice of their family members, or the voice of celebrities, etc., and use them as model voices to sing with an electronic musical instrument. In this case, it is also possible to achieve a singing performance that is particularly natural and has high sound quality compared to the past as a lower-priced electronic musical instrument.

[0096] (Lyrics Progression Control Method)

[0097] The following describes a method for controlling the progression of lyrics according to an embodiment of the present disclosure. In addition, the lyrics progression control of the present disclosure can also be replaced with performance control, performance, etc.

[0098] The action subject (electronic musical instrument 10) of each of the following flowcharts may also be replaced by any one of the CPU 201, the waveform data output unit 211 (or its internal sound source LSI 204, the sound synthesis LSI 205 (processing unit 306, singing control unit 307, sound source 308, singing synthesis unit 309, mute unit 310, etc.)) or a combination thereof. For example, the CPU 201 may execute the control program loaded from the ROM 202 to the RAM 203 to implement each action.

[0099] In addition, at the beginning of the process shown below, initialization processing may also be performed. This initialization processing may also include interrupt processing, lyrics progression, export of tick times that serve as reference times for automatic accompaniment, rhythm setting, song selection, song reading, instrument sound selection, and other processing associated with buttons, etc.

[0100] The CPU 201 can detect the operation of the switch panel 140 b , the keyboard 140 k , the pedal 140 p , etc. at appropriate timing based on the interrupt from the key scanner 206 , and perform corresponding processing.

[0101] In addition, the following is an example of controlling the progression of lyrics, but the object of progression control is not limited to this. Based on the present disclosure, for example, the progression of any character string, article (such as a news article), etc. can also be controlled instead of the lyrics. That is, the lyrics of the present disclosure can also be replaced with characters, character strings, etc.

[0102] In the present disclosure, the electronic musical instrument 10 generates the singing voice waveform data 217 (sound synthesis data) regardless of the user's performance operation, and controls whether to allow or not to allow the sound corresponding to the singing voice waveform data 217 to be emitted.

[0103] For example, the electronic musical instrument 10 can also generate singing waveform data 217 (sound synthesis data) in real time according to the singing data 215 (which may or may not be stored in the memory before the performance starts) based on the indication of the start of the performance, even if the key pressed by the user is not detected.

[0104] The electronic musical instrument 10 performs a mute process so that the sound corresponding to the real-time generated singing waveform data 217 (sound synthesis data) is not emitted during the period when a key is not detected (the user cannot hear the singing). In addition, when a key is detected, the electronic musical instrument 10 cancels the mute process (the user can hear the singing). The electronic musical instrument 10 does not perform a mute process on the song waveform data 218 (the user can hear the accompaniment when the user cannot hear the singing).

[0105] When the user presses a key, the electronic musical instrument 10 overwrites the pitch data corresponding to the key timing in the singing data 215 (hereinafter, sometimes also referred to as singing data) with the pitch data corresponding to the pressed key. Thus, singing waveform data 217 (hereinafter, sometimes also referred to as singing waveform data) is generated based on the overwritten pitch data. In addition, the electronic musical instrument 10 can also perform singing reproduction processing regardless of whether the mute processing is performed or not.

[0106] In other words, the processor of the electronic musical instrument 10 can also generate the singing voice synthesis data 217 according to the singing voice data 215 in both the case where the user operation (pressing the key) on the performance operating member (key) is detected and the case where no operation is detected. In addition, the processor of the electronic musical instrument 10 is controlled in the following manner: when the user operation on the performance operating member is detected, the singing voice according to the generated singing voice synthesis data is allowed to be emitted, and when the user operation on the performance operating member is not detected at all, the singing voice according to the generated singing voice synthesis data is not allowed to be emitted.

[0107] According to such a configuration, the presence or absence of the synthesized sound automatically reproduced in the background can be controlled by the key operation of the user as a trigger, so the part of the lyrics that the user wants to pronounce can be easily specified.

[0108] Furthermore, the processor of the electronic musical instrument 10 changes the singing voice data according to the passage of time in both cases where the user operation on the performance operating element is detected and the user operation is not detected. With such a configuration, the lyrics played in the background can be appropriately shifted.

[0109] When the user operation is detected, the processor of the electronic musical instrument 10 may instruct to produce the singing voice according to the generated singing voice synthesis data at the pitch specified by the user operation. According to such a configuration, the pitch of the synthesized sound produced can be easily changed.

[0110] The processor of the electronic musical instrument 10 can also instruct to mute the singing voice issued according to the generated singing voice synthesis data when the user operation is not detected at all. According to such a structure, the synthesized sound can be heard when it is not needed, and the pronunciation can be switched at high speed when it is needed.

[0111] Figure 6 This is a diagram showing an example of a flowchart of a lyrics progression control method according to one embodiment.

[0112] First, the electronic musical instrument 10 reads the song data and the singing voice data (step S101). Figure 4 , Figure 5 The singing data 215) may also be singing data corresponding to the song data.

[0113] The electronic musical instrument 10 starts to play the song data corresponding to the lyrics (in other words, the reproduction of the accompaniment) according to the user's operation (step S102). The user can perform key operations in accordance with the accompaniment.

[0114] The electronic musical instrument 10 starts counting up the lyrics pronunciation timing t (step S103 ). The electronic musical instrument 10 may process the t in at least one unit such as beat, tick, or second. The lyrics pronunciation timing t may also be counted by the timer 210 .

[0115] The electronic musical instrument 10 substitutes 1 into the lyrics index (also indicated as "n") indicating the position of the lyrics to be played next (step S104). In addition, when starting the lyrics midway (for example, starting from the last storage position), a value other than 1 may be substituted into n.

[0116] The lyrics index can also be a variable that indicates the syllable (or character) corresponding to the first syllable (or character) when the lyrics are regarded as a string. For example, the lyrics index n can also be represented by Figure 4 , Figure 5 The singing voice data (n-th lyrics data) at the n-th singing voice reproduction position shown in the figure above.

[0117] In the present disclosure, a lyric corresponding to a position of one lyric (lyric index) may correspond to one or more characters constituting one syllable. The syllables included in the singing voice data may include various syllables such as vowels only, consonants only, and consonants + vowels.

[0118] In addition, the electronic musical instrument 10 stores lyrics pronunciation timing tn corresponding to lyrics index n (n=1, 2, ..., N) based on the start of song data pronunciation (the beginning of accompaniment). Here, N is equivalent to the last lyrics. Lyrics pronunciation timing tn can represent the desired timing of the nth singing voice reproduction position.

[0119] The electronic musical instrument 10 determines whether the lyrics sounding timing t is the nth timing (in other words, whether t=tn) (step S105). If t=tn (step S105-Yes), the electronic musical instrument 10 determines whether a key is pressed (a note-on event occurs) (step S106).

[0120] When a key is pressed (step S106 —Yes), the electronic musical instrument 10 overwrites the pitch data of the n-th lyrics data (the pitch data of the read singing voice data) with the pitch data corresponding to the pressed key (step S107 ).

[0121] The electronic musical instrument 10 generates singing waveform data based on the pitch data and the nth lyrics data (characters of the nth lyrics) overwritten in step S107 (step S108). The electronic musical instrument 10 performs a sound production process based on the singing waveform data produced in step S108 (step S109). Unless a mute process is performed in step S112 described later, the sound production process may be a process of producing only the duration of the nth lyrics data.

[0122] In step S109, it is also possible to Figure 4 The electronic musical instrument 10 may, for example, obtain the acoustic feature data (formant information) of the n-th singing voice data through the singing voice control unit 307, instruct the sound source 308 to produce a musical instrument sound of a pitch corresponding to the key press (generation of musical instrument sound waveform data), and instruct the singing voice synthesis unit 309 to add the formant information of the n-th singing voice data to the musical instrument sound waveform data output from the sound source 308.

[0123] In step S109, the electronic musical instrument 10, for example, the processing unit 306 inputs the designated pitch data (pitch data corresponding to the pressed key) and the n-th singing voice data (n-th lyrics data) to the singing voice control unit 307, and the singing voice control unit 307 outputs the corresponding formant information 318 and vocal cord sound source data (pitch information) 319 to the singing voice synthesis unit 309 based on the input estimated acoustic feature quantity sequence 317. The singing voice synthesis unit 309 generates the n-th singing voice waveform data (which can also be called the singing voice waveform data of the n-th lyrics corresponding to the n-th note) based on the input formant information 318 and vocal cord sound source data (pitch information) 319, and outputs it to the sound source 308. In this way, the sound source 308 obtains the n-th singing voice waveform data from the singing voice synthesis unit 309 and performs sound production processing on the data.

[0124] In step S109, it is also possible to Figure 5The processing unit 307 of the electronic musical instrument 10 inputs the designated pitch data (pitch data corresponding to the pressed key) and the n-th singing voice data (n-th lyrics data) to the singing voice control unit 306. Then, the singing voice control unit 306 of the electronic musical instrument 10 estimates the acoustic feature quantity sequence 317 based on the input, and outputs the corresponding formant information 318 and vocal cord sound source data (pitch information) 319 to the singing voice synthesis unit 309.

[0125] Furthermore, the singing voice synthesis unit 309 generates n-th singing voice waveform data (which may also be referred to as singing voice waveform data of n-th lyrics corresponding to n-th notes) based on the input formant information 318 and vocal cord sound source data (pitch information) 319, and outputs it to the sound source 308. Then, the sound source 308 obtains the n-th singing voice waveform data from the singing voice synthesis unit 309. The electronic musical instrument 10 performs sound production processing based on the sound source 308 on the obtained n-th singing voice waveform data.

[0126] In addition, other pronunciation processing in the flowchart can also be performed in the same manner.

[0127] After step S109, the electronic musical instrument 10 increments n by 1 (substitutes n+1 into n) (step S110).

[0128] The electronic musical instrument 10 determines whether all keys are released (step S111). If all keys are released (step S111-Yes), the electronic musical instrument 10 performs a mute process of the sound corresponding to the singing waveform data (step S112). The mute process can be performed by the mute unit 310 described above.

[0129] After step S112 or step S111-No, the electronic musical instrument 10 determines whether the reproduction of the song data started in step S102 has ended (step S113). If it has ended (step S113-Yes), the electronic musical instrument 10 may also end the processing of this flowchart and return to the standby state. Otherwise (step S113-No), it returns to step S105.

[0130] In addition, if no key is pressed after step S105-Yes (step S106-No), the electronic musical instrument 10 generates singing waveform data based on the pitch data of the nth lyrics data (uncovered pitch data) and the character data of the nth lyrics data (step S114). The electronic musical instrument 10 performs a mute process based on the pronunciation of the singing waveform data generated in step S114 (step S115), and proceeds to step S110.

[0131] In addition, when t<tn (step S105-No), the electronic musical instrument 10 determines whether there is a key being sounded (for example, there is a sound produced based on step S109, and there is a key being pressed) (step S116). When there is a key being sounded (step S116-Yes), the electronic musical instrument 10 changes the pitch of the sound being sounded (step S117), and returns to step S105.

[0132] For example, the pitch may be changed by generating singing waveform data based on the pitch data corresponding to the pressed key and the lyrics being pronounced (character data of the n-1th lyrics data) and performing pronunciation processing, as in the case described in steps S107-S109. If there is no key being pronounced (step S116-No), the process returns to step S105.

[0133] In addition, step S116 may be a determination of whether a key is pressed, regardless of whether the key is being pronounced. In this case, step S117 may be the release of the muting process of steps S112, S115, etc. (in other words, for the muted sound, the pronunciation process of the key is performed).

[0134] In addition, when the keys pressed in steps S106, S116, etc. are multiple keys pressed simultaneously (keys pressed for harmony), singing sounds (polyphony) with harmonics corresponding to the respective pitches can also be emitted through steps S107-S109, S117, etc.

[0135] In this flowchart, by applying the mute processing instead of the silencing processing in steps S112, S115, etc., the sound is reproduced in the background even when it is not emitted, so that the sound can be emitted quickly when it is desired.

[0136] Figure 7 1 is a diagram showing an example of lyrics progression controlled by a lyrics progression control method according to an embodiment. In this example, an example of a performance corresponding to the illustrated music score is described. It is assumed that lyrics indexes 1 to 6 correspond to "Sle", "ep", "in", "heav", "en", and "ly", respectively.

[0137] In this example, the electronic musical instrument 10 determines that a key is pressed by the user at timing t1 corresponding to the lyrics index 1. Figure 7 In this case, the electronic musical instrument 10 overwrites the pitch data corresponding to the lyrics index 1 with the pitch data corresponding to the pressed key, and emits the lyrics "S1e" (steps S107-S109). At this time, the electronic musical instrument 10 does not apply the mute process.

[0138] The electronic musical instrument 10 determines that no key is pressed by the user at timings t2 and t3 corresponding to the lyrics indexes 2 and 3. In this case, the electronic musical instrument 10 generates singing waveform data of the lyrics "ep" and "in" corresponding to the lyrics indexes 2 and 3, and performs a mute process (steps S114-S115). Therefore, although the user cannot hear the singing of the lyrics "ep" and "in", he can hear the accompaniment.

[0139] In addition, the electronic musical instrument 10 determines that a key is pressed by the user at timing t4 corresponding to lyrics index 4. In this case, the electronic musical instrument 10 overwrites the pitch data corresponding to lyrics index 4 with the pitch data corresponding to the pressed key, and pronounces the lyrics "heav". At this time, the electronic musical instrument 10 does not apply the mute process.

[0140] The electronic musical instrument 10 determines that no key is pressed by the user at timings t5 and t6 corresponding to the lyrics indexes 5 and 6. In this case, the electronic musical instrument 10 generates singing waveform data of the lyrics "en" and "ly" corresponding to the lyrics indexes 5 and 6, and performs a mute process. Therefore, although the user cannot hear the singing of the lyrics "en" and "ly", he can hear the accompaniment.

[0141] That is, according to the lyrics progression control method of one embodiment of the present disclosure, depending on the user's performance method, sometimes part of the lyrics are silent (in Figure 7 In the example above, sometimes the “epin” between “Sle” and “heav” is silent).

[0142] The usual automatic playing automatically plays the lyrics even without the user pressing a key (in the above Figure 7 In the example of , "Sleep in heavenly" is pronounced in full, and the pitch cannot be changed). In contrast, according to the above-mentioned lyrics progression control method, the lyrics can be automatically played only when a key is pressed (and the pitch can also be changed).

[0143] Additionally, the lyrics advance with each current key press (when applied to Figure 7 In the case of a technique in which the lyrics index is incremented and pronounced each time a key is pressed, if the position of the lyrics exceeds the expected position due to excessive key pressing or if the position of the lyrics does not progress as expected due to insufficient key pressing, a synchronization process (a process for matching the position of the lyrics with the playback position of the accompaniment) is required to appropriately move the position of the lyrics. On the other hand, according to the above-mentioned lyrics progress control method, such a synchronization process is not required, and an increase in the processing load of the electronic musical instrument 10 is appropriately suppressed.

[0144] (Variation Example)

[0145] Figure 4 , Figure 5The sound synthesis processing shown in the example above can be switched on / off based on the user's operation of the switch panel 140b. When it is off, the waveform data output unit 211 can be controlled to generate and output a sound source signal of musical instrument sound data of a pitch corresponding to the key.

[0146] exist Figure 6 In the flowchart, some steps may be omitted. When the determination process is omitted, the determination may be interpreted as a route that always enters "yes" or always enters "no" in the flowchart.

[0147] The electronic musical instrument 10 may also control the display 150d to display lyrics. For example, the lyrics near the current lyrics position (lyrics index) may be displayed, or the lyrics corresponding to the sounds being pronounced or the lyrics corresponding to the sounds being pronounced may be displayed in color, etc., so that the current lyrics position can be identified.

[0148] The electronic musical instrument 10 may transmit at least one of the singing voice data and information about the current position of lyrics to the external device. The external device may control its own display to display the lyrics based on the received singing voice data and information about the current position of lyrics.

[0149] In the above example, the electronic musical instrument 10 is a keyboard instrument such as a keyboard, but the present invention is not limited thereto. The electronic musical instrument 10 may be any device having a structure that allows the timing of sound generation to be specified by a user's operation, and may be an electric violin, electric guitar, drum, trumpet, or the like.

[0150] Therefore, the "key" of the present disclosure can also be replaced by strings, valves, other performance operating parts for specifying pitches, arbitrary performance operating parts, etc. The "key pressing" of the present disclosure can also be replaced by keystrokes, picking, performance, operation of operating parts, etc. The "key release" of the present disclosure can also be replaced by stopping the strings, stopping the performance, stopping the operating parts (non-operation), etc.

[0151] In addition, the block diagram used in the description of the above-mentioned embodiment represents a block of a functional unit. These functional blocks (structural parts) are implemented by any combination of hardware and / or software. In addition, the implementation unit of each functional block is not particularly limited. That is, each functional block can be implemented by a physically combined device, or by connecting two or more physically separated devices by wire or wireless, and implemented by these multiple devices.

[0152] Furthermore, the terms described in the present disclosure and / or the terms necessary for understanding the present disclosure may be replaced with terms having the same or similar meanings.

[0153] The information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values ​​to a specified value, or other corresponding information. In addition, the names used in parameters, etc. in this disclosure are not limited in any way.

[0154] The information, signals, etc. described in the present disclosure may also be represented using any of a variety of different technologies. For example, data, commands, instructions, information, signals, bits, symbols, chips, etc. that may be mentioned in the above description as a whole may also be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, light fields or photons, or any combination thereof.

[0155] Information, signals, etc. may also be input and output via multiple network nodes. The input and output information, signals, etc. may be stored in a specific location (e.g., a memory) or managed using a table. The input and output information, signals, etc. may be overwritten, updated, or rewritten. The output information, signals, etc. may also be deleted. The input information, signals, etc. may also be sent to other devices.

[0156] Software shall be construed broadly to mean not only software, firmware, middleware, microcode, hardware description language or other names, but also commands, sets of commands, codes, code segments, program code, programs, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, executables, execution threads, steps, functions, etc.

[0157] In addition, software, commands, information, etc. may also be sent and received via a transmission medium. For example, when the software is sent from a website, server, or other remote source using at least one of wired technology (coaxial cable, optical cable, twisted pair, digital subscriber line (DSL), etc.) and wireless technology (infrared, microwave, etc.), at least one of these wired technologies and wireless technologies is included in the definition of transmission medium.

[0158] Each method / implementation method described in this disclosure may be used alone, in combination, or switched during execution. In addition, the processing order, timing, flow chart, etc. of each method / implementation method described in this disclosure may be swapped as long as there is no contradiction. For example, regarding the method described in this disclosure, the elements of various steps are presented in an illustrative order, and are not limited to the specific order presented.

[0159] The term “based on” used in the present disclosure does not mean “based only on” unless otherwise specified. In other words, the term “based on” means both “based only on” and “based at least on”.

[0160] Any reference to elements using the terms "first", "second", etc. used in this disclosure does not limit the amount or order of these elements as a whole. These terms can be used in this disclosure as a convenient method to distinguish between two or more elements. Therefore, the reference to the first and second elements does not mean that only two elements can be used or that the first element must precede the second element in some form.

[0161] In the present disclosure, when the terms “include,” “including,” and variations thereof are used, these terms mean inclusion, similar to the term “comprising.” In addition, the term “or” used in the present disclosure does not mean exclusive OR.

[0162] In the present disclosure, for example, when an article is added due to translation, the present disclosure may also include a case where the noun following the article is in a plural form.

[0163] The invention of the present disclosure has been described in detail above, but it is obvious to those skilled in the art that the invention of the present disclosure is not limited to the embodiments described in the present disclosure. The invention of the present disclosure can be implemented as a modified and altered mode without departing from the gist and scope of the invention determined based on the description of the claims. Therefore, the description of the present disclosure is for the purpose of illustrative description and has no limiting meaning for the invention of the present disclosure.

Claims

1. An electronic musical instrument, wherein: have: performance operating elements; and At least 1 processor, The at least one processor is controlled in the following manner: When the user operation on the performance operating member is not detected at a timing when the user operation should be detected, first singing voice synthesis data is generated according to lyrics data stored before the performance corresponding to the timing, When the user operation is detected at a timing when the user operation should be detected, the pitch data of the lyrics data stored before the performance is overwritten with the pitch data specified by the user operation, and second singing voice synthesis data is generated based on the overwritten pitch data. When the user operation is detected at the timing, the singing voice according to the generated second singing voice synthesis data is allowed to be emitted, When the user operation is not detected at the timing, the singing voice according to the generated first singing voice synthesis data is not allowed to be uttered.

2. The electronic musical instrument according to claim 1, wherein: When the user operation is not detected at the timing, the at least one processor generates the first singing voice synthesis data according to the pitch indicated by the pitch data included in the lyrics data.

3. The electronic musical instrument according to claim 1 or 2, wherein: The at least one processor instructs muting of the singing voice uttered according to the generated first singing voice synthesis data when the user operation is not detected at the timing.

4. The electronic musical instrument according to claim 1 or 2, wherein: The at least one processor, Instructs to play the accompaniment corresponding to the song data, When the user operation is not detected at the timing, the singing voice according to the generated first singing voice synthesis data is not allowed to be emitted, and the accompaniment is continued to be emitted.

5. The electronic musical instrument according to claim 1 or 2, wherein: The electronic musical instrument includes a memory storing a trained model that has learned acoustic feature quantities of a singer's singing voice. The at least one processor generates the second singing voice synthesis data according to the acoustic feature data output by the trained model in response to the input of the lyrics data into the trained model in accordance with the user operation.

6. The electronic musical instrument according to claim 1 or 2, wherein: The lyrics data includes first character data corresponding to a first timing, second character data corresponding to a second timing after the first timing, and third character data corresponding to a third timing after the second timing. The at least one processor, Based on the detection of the user operation corresponding to the first timing, instructing to emit a singing sound corresponding to the first character data, When the user operation corresponding to the second timing is not detected but the user operation corresponding to the third timing is detected, the singing of the song corresponding to the third character data is instructed instead of the singing of the song corresponding to the second character data.

7. The electronic musical instrument according to claim 1 or 2, wherein: The at least one processor, Based on the detection of the user operation corresponding to the first timing, an instruction is given to emit a singing sound corresponding to the first character data, wherein the first character data is the first character data in the lyrics data including the first character data corresponding to the first timing, the second character data corresponding to the second timing after the first timing, and the third character data corresponding to the third timing after the second timing, When the user operation corresponding to the second timing is not detected but the user operation corresponding to the third timing is detected, the singing of the song corresponding to the third character data is instructed instead of the singing of the song corresponding to the second character data.

8. A method for controlling an electronic musical instrument, wherein: At least one processor of the electronic musical instrument is controlled in the following manner: When the user operation on the performance operating member of the electronic musical instrument is not detected at a timing when the user operation should be detected, first singing voice synthesis data is generated according to the lyrics data stored before the performance corresponding to the timing, When the user operation is detected at a timing when the user operation should be detected, the pitch data of the lyrics data stored before the performance is overwritten with the pitch data specified by the user operation, and second singing voice synthesis data is generated based on the overwritten pitch data. When the user operation is detected at the timing, the singing voice according to the generated second singing voice synthesis data is allowed to be emitted, When the user operation is not detected at the timing, the singing voice according to the generated first singing voice synthesis data is not allowed to be uttered.

9. The control method of an electronic musical instrument according to claim 8, wherein: When the user operation is not detected at the timing, the at least one processor generates the first singing voice synthesis data according to the pitch indicated by the pitch data included in the lyrics data.

10. The control method of an electronic musical instrument according to claim 8 or 9, wherein: The at least one processor instructs muting of the singing voice uttered according to the generated first singing voice synthesis data when the user operation is not detected at the timing.

11. The control method of an electronic musical instrument according to claim 8 or 9, wherein: The at least one processor, Instructs to play the accompaniment corresponding to the song data, When the user operation is not detected at the timing, the singing voice according to the generated first singing voice synthesis data is not allowed to be emitted, and the accompaniment is continued to be emitted.

12. The control method of an electronic musical instrument according to claim 8 or 9, wherein: The electronic musical instrument includes a memory storing a trained model that has learned acoustic feature quantities of a singer's singing voice. The at least one processor generates the second singing voice synthesis data according to the acoustic feature data output by the trained model in response to the input of the lyrics data into the trained model in accordance with the user operation.

13. The control method of an electronic musical instrument according to claim 8 or 9, wherein: The lyrics data includes first character data corresponding to a first timing, second character data corresponding to a second timing after the first timing, and third character data corresponding to a third timing after the second timing. The at least one processor, Based on the detection of the user operation corresponding to the first timing, instructing to emit a singing sound corresponding to the first character data, When the user operation corresponding to the second timing is not detected but the user operation corresponding to the third timing is detected, the singing of the song corresponding to the third character data is instructed instead of the singing of the song corresponding to the second character data.

14. The control method of an electronic musical instrument according to claim 8 or 9, wherein: The at least one processor, Based on the detection of the user operation corresponding to the first timing, an instruction is given to emit a singing sound corresponding to the first character data, wherein the first character data is the first character data in the lyrics data including the first character data corresponding to the first timing, the second character data corresponding to the second timing after the first timing, and the third character data corresponding to the third timing after the second timing, When the user operation corresponding to the second timing is not detected but the user operation corresponding to the third timing is detected, the singing of the song corresponding to the third character data is instructed instead of the singing of the song corresponding to the second character data.

Images