Information processing device, electronic musical instrument, electronic musical instrument system, method and program

Abstract

To provide an information processing device, an electronic musical instrument, and an electronic musical instrument system, a method, and a program, capable of producing a further natural voice with a smaller memory capacity in response to an operation of the electronic musical instrument.SOLUTION: In the electronic musical instrument, in response to a detection of a key-press operation of a keyboard, a CPU causes a syllable Come based on a singing voice parameter corresponding to a syllable start frame to be pronounced. Then, if a state in which any key of the keyboard is being pressed continues after a pronunciation of a vowel ah based on the singing voice parameter corresponding to a certain vowel frame in a vowel section included in the syllable is started, the pronunciation of the vowel based on the singing voice parameter corresponding to the certain vowel frame is continued until the operation of the pressed key is cancelled (i.e., key release).SELECTED DRAWING: Figure 12

Description

[Technical field]

[0001] The present invention relates to an information processing device, an electronic musical instrument, an electronic musical instrument system, a method, and a program. [Background technology]

[0002] 2. Description of the Related Art Conventionally, there is known a technique for reproducing lyrics syllable by syllable in response to depression of a key on an electronic musical instrument such as a keyboard instrument. For example, Patent Document 1 describes an audio information reproduction method that reads out audio information in which waveform data of each of a plurality of vocalization units, each of which has a determined pronunciation pitch and pronunciation order, is time-series, reads out delimiter information associated with the audio information, the delimiter information defining a playback start position, a loop start position, a loop end position and a playback end position for each vocalization unit, moves the playback position in the audio information based on the delimiter information in response to obtaining note-on information or note-off information, and starts playback from the loop end position to the playback end position of the vocalization unit to be reproduced in response to obtaining note-off information corresponding to the note-on information. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2020 / 217801 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in Patent Document 1, since audio information, which is waveform data of a plurality of vocalization units, is connected to perform pronunciation for each syllable or loop playback, it is difficult to produce a natural singing voice. Also, since it is necessary to store audio information in which the waveform data of each of a plurality of vocalization units is time-series, a large memory capacity is required.

[0005] The present invention has been made in consideration of the above problems, and has an object to make it possible to produce more natural sounds in response to the operation of an electronic musical instrument while requiring less memory capacity. [Means for solving the problem]

[0006] In order to solve the above problems, the information processing device of the present invention comprises: a control unit which starts pronunciation of a syllable based on parameters corresponding to a syllable start frame in response to detection of an operation on an operator, and then, when the operation on the operator continues even after the start of pronunciation of a vowel based on parameters corresponding to a certain vowel frame in a vowel section included in the syllable, continues pronouncing the vowel based on the parameters corresponding to the certain vowel frame until the operation on the operator is released; Equipped with The control unit is when the operation of the operator is released, if the next frame position to be sounded exceeds the vowel end position and no operation of any operator is detected, it is determined whether or not the next frame position exceeds the syllable end position; If it does not exceed the threshold, a pronunciation process for the consonant is executed based on the parameters corresponding to the consonant frame following the vowel frame, and a mute process is executed for the pronunciation process for the consonant. Effect of the Invention

[0007] According to the present invention, it is possible to produce more natural sounds in response to the operation of an electronic musical instrument with a smaller memory capacity. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram showing an example of the overall configuration of an electronic musical instrument system according to the present invention; [Diagram 2] FIG. 2 is a diagram showing the external appearance of the electronic musical instrument shown in FIG. [Diagram 3] 2 is a block diagram showing a functional configuration of the electronic musical instrument shown in FIG. 1. [Figure 4] 2 is a block diagram showing a functional configuration of the terminal device of FIG. 1. [Diagram 5] 2 is a diagram showing a configuration related to the generation of singing voices in response to key depression operations on a keyboard in a singing voice generation mode of the electronic musical instrument of FIG. 1. FIG. [Figure 6] FIG. 1 is an image diagram showing the relationship between frames and syllables. [Figure 7] 4 is a flowchart showing the flow of a singing voice pronunciation mode process executed by the CPU of FIG. 3. [Figure 8] 4 is a flowchart showing the flow of a voice synthesis process A executed by the CPU of FIG. 3. [Figure 9] 4 is a flowchart showing the flow of a voice synthesis process B executed by the CPU of FIG. 3. [Figure 10] 4 is a flowchart showing the flow of a voice synthesis process C executed by the CPU of FIG. 3. [Figure 11] 4 is a flowchart showing the flow of a voice synthesis process D executed by the CPU of FIG. [Figure 12] FIG. 13 is a graph showing the change in volume from when a key is pressed to when a key is released and the volume becomes 0, when the syllable "Come" is sounded in response to keyboard operation in singing voice sound production mode processing, and is also showing a schematic diagram of the frame positions used for sound production at each timing of the graph. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments of the present invention will be described with reference to the drawings. However, the embodiments described below are subject to various limitations that are technically preferable for implementing the present invention. Therefore, the technical scope of the present invention is not limited to the following embodiments and illustrated examples.

[0010] [Configuration of Electronic Musical Instrument System 1] FIG. 1 is a diagram showing an example of the overall configuration of an electronic musical instrument system 1 according to the present invention. As shown in FIG. 1, an electronic musical instrument system 1 includes an electronic musical instrument 2 and a terminal device 3 connected via a communication interface I (or a communication network N).

[0011] [Electronic Instrument 2 Configuration] The electronic musical instrument 2 has a normal mode in which it outputs musical instrument sounds in response to the user's key depression on the keyboard 101, as well as a singing voice generation mode in which it generates singing voices in response to the user's key depression on the keyboard 101. In this embodiment, the electronic musical instrument 2 has a first mode and a second mode as singing voice generation modes. The first mode is a mode for generating a singing voice that faithfully reproduces the voice of a human (singer). The second mode is a mode for generating a singing voice with a tone color that combines a set tone color (such as a musical instrument sound) and a human singing voice.

[0012] 2 is a diagram showing an example of the external appearance of the electronic musical instrument 2. The electronic musical instrument 2 includes a keyboard 101 consisting of multiple keys as operators (performance operators), a switch panel 102 for instructing various settings, a parameter change operator 103, and an LCD 104 (Liquid Crystal Display) for various displays. The electronic musical instrument 2 also includes a speaker 214 on the rear, side, or back portion, etc., for emitting musical tones and voices (singing voices) generated by playing.

[0013] Fig. 3 is a block diagram showing the functional configuration of the control system of the electronic musical instrument 2 of Fig. 1. As shown in Fig. 3, the electronic musical instrument 2 is configured by connecting a CPU (Central Processing Unit) 201 connected to a timer 210, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, a sound source section 204, a voice synthesis section 205, an amplifier 213, a key scanner 206 connected to the keyboard 101, switch panel 102, and parameter change operator 103 of Fig. 2, an LCD controller 207 connected to the LCD 104 of Fig. 2, and a communication section 208 to a bus 209. In this embodiment, the switch panel 102 includes a singing voice sound generation mode switch, a first mode / second mode change switch, and a tone color setting switch, which will be described later.

[0014] In addition, D / A converters 211 and 212 are connected to the sound source unit 204 and the voice synthesis unit 205, respectively, and the waveform data of the musical instrument sound output from the sound source unit 204 and the voice waveform data of the singing voice (singing voice waveform data) output from the voice synthesis unit 205 are converted into analog signals by the D / A converters 211 and 212, respectively, amplified by an amplifier 213, and then output (i.e., pronounced) from a speaker 214.

[0015] The CPU 201 executes a program stored in the ROM 202 while using the RAM 203 as a work memory, thereby executing the control operation of the electronic musical instrument 2 in Fig. 1. The CPU 201 executes a singing voice sound generation mode process, which will be described later, in cooperation with the program stored in the ROM 202, thereby realizing the function of the control unit of the information processing device of the present invention. The ROM 202 stores programs and various fixed data.

[0016] The sound source unit 204 has a waveform ROM in which waveform data of various tones such as human voice, dog voice, cat voice, etc. are stored as waveform data for vocal sound source in singing voice pronunciation mode (vocal sound source waveform data) in addition to waveform data of musical instrument sounds such as piano, organ, synthesizer, string instruments, and wind instruments (musical instrument sound waveform data). Note that the musical instrument sound waveform data can also be used as vocal sound source waveform data.

[0017] In the normal mode, the sound source unit 204 reads out musical instrument sound waveform data, for example, from a waveform ROM (not shown) based on pitch information of the pressed key of the keyboard 101 in accordance with a control instruction from the CPU 201, and outputs the data to the D / A converter 211. In the second mode of the singing voice sound mode, the sound source unit 204 reads out waveform data, for example, from a waveform ROM (not shown) based on pitch information of the pressed key of the keyboard 101 in accordance with a control instruction from the CPU 201, and outputs the waveform data to the voice synthesis unit 205 as vocal sound source waveform data. The sound source unit 204 can output waveform data for multiple channels at the same time. It is also possible to generate waveform data corresponding to the pitch of the pressed key of the keyboard 101 based on the pitch information and the waveform data stored in the waveform ROM. The sound source unit 204 is not limited to a PCM (Pulse Code Modulation) sound source format, and may use other sound source formats such as an FM (Frequency Modulation) sound source format.

[0018] The voice synthesis unit 205 has a sound source generation unit and a synthesis filter, and generates singing voice waveform data based on pitch information and singing voice parameters given by the CPU 201, or based on singing voice parameters given by the CPU 201 and vocal sound source waveform data input from the sound source unit 204, and outputs the data to the D / A converter 212.

[0019] In addition, the sound source unit 204 and the voice synthesis unit 205 may be configured using dedicated hardware such as an LSI (Large-Scale Integration) or may be realized by software in cooperation with the CPU 201 and a program stored in the ROM 202.

[0020] The key scanner 206 constantly scans the key on / key off states of each key on the keyboard 101 in FIG. 2 and the operation states of the switch panel 102 and the parameter change operators 103, and outputs the pitch and key on / key off information (performance operation information) of the operated keys on the keyboard 101 and operation information of the switch panel 102 and the parameter change operators 103 to the CPU 201. Here, the parameter change operator 103 is a switch for the user to set (instruct to change) the timbre (tone) of the singing voice to be pronounced in the singing voice pronunciation mode. As shown in FIG. 2, the parameter change operator 103 of this embodiment is configured to be rotatable within a range in which the position of the indicator 103a is between scales 1 and 2, and the tone of the singing voice to be pronounced in the singing voice pronunciation mode can be set (changed) between a first voice and a second voice different from the first voice according to the position of the indicator 103a. For example, by turning the parameter change operator 103 clockwise to the maximum (for example, the indicator 103a is set to scale 1), the tone of the singing voice to be pronounced in the singing voice pronunciation mode can be set to the first voice (for example, a male voice). By turning the parameter change operator 103 counterclockwise to the maximum (for example, the indicator 103a is set to scale 2), the tone of the singing voice to be pronounced in the singing voice pronunciation mode can be set to the second voice (for example, a female voice). Furthermore, by positioning the indicator 103a of the parameter change operator 103 between scale 1 and scale 2, it is possible to set the voice tone to be a combination of the first voice and the second voice. The ratio at which the first voice and the second voice are combined is determined according to the ratio of the rotation angle from scale 1 to the rotation angle from scale 2.

[0021] The LCD controller 207 is an integrated circuit (IC) that controls the display state of the LCD 104 . A communication unit 208 transmits and receives data to and from an external device such as a terminal device 3 connected via a communication network N such as the Internet or a communication interface I such as a USB (Universal Serial Bus) cable.

[0022] [Configuration of terminal device 3] FIG. 4 is a block diagram showing a functional configuration of the terminal device 3 of FIG. 4, the terminal device 3 is a computer including a CPU 301, a ROM 302, a RAM 303, a storage unit 304, an operation unit 305, a display unit 306, a communication unit 307, etc., and each unit is connected via a bus 308. As the terminal device 3, for example, a tablet PC (Personal Computer), a notebook PC, a smartphone, etc. can be used.

[0023] The ROM 302 of the terminal device 3 is equipped with a trained model 302a and a trained model 302b. The trained model 302a and the trained model 302b are generated by machine learning a plurality of data sets consisting of musical score data (lyric data (text information of lyrics) and pitch data (including information on the length of notes)) of a plurality of songs and singing waveform data when a singer (human) sings each of the songs. The trained model 302a is generated by machine learning the singing waveform data of a first singer (e.g., a male) corresponding to the above-mentioned first voice. The trained model 302b is generated by machine learning the singing waveform data of a second singer (e.g., a female) corresponding to the above-mentioned second voice. When lyric data and pitch data of an arbitrary song (or a phrase) are input, the trained model 302a and the trained model 302b each infer a group of singing voice parameters (called singing voice information) for producing a singing voice equivalent to that produced when the trained model was generated and the singer sang the input song.

[0024] [Singing voice pronunciation mode operation] Fig. 5 is a diagram showing a configuration related to the generation of a singing voice in the singing voice generation mode in response to key depression of the keyboard 101. Hereinafter, with reference to Fig. 5, an operation when the electronic musical instrument 2 generates a singing voice in response to key depression of the keyboard 101 in the singing voice generation mode will be described.

[0025] When the user wishes to perform in the singing voice mode, the user presses the singing voice mode switch on the switch panel 102 of the electronic musical instrument 2 to instruct a transition to the singing voice mode. When the singing voice pronunciation mode switch is pressed, the CPU 201 shifts the operation mode to the singing voice pronunciation mode. In addition, in response to pressing of the first mode / second mode changeover switch of the switch panel 102, the CPU 201 switches between the first mode and the second mode in the singing voice pronunciation mode. When the second mode is set, and the user selects the tone color of the voice to be generated by using the tone color selection switch of the switch panel 102, the CPU 201 sets the information of the selected tone color in the sound source unit 204.

[0026] Next, the user inputs, into the terminal device 3, lyric data and pitch data of a desired song that the user wishes to have the electronic musical instrument 2 produce in the singing voice production mode, using a dedicated application or the like. The lyric data and pitch data of a desired song may be stored in the storage unit 304, and the lyric data and pitch data of a desired song may be selected from those stored in the storage unit 304. When lyric data and pitch data of a song to be pronounced in singing voice pronunciation mode are input into the terminal device 3, the CPU 301 inputs the lyric data and pitch data of the input song into the trained model 302a and the trained model 302b, causes each to infer a group of singing voice parameters, and transmits the inferred group of singing voice parameters, which is the singing voice information, to the electronic musical instrument 2 via the communication unit 307.

[0027] Here, the singing voice information will be described. Each section obtained by dividing a song into a predetermined time unit in the time direction is called a frame, and the trained model 302a and the trained model 302b generate singing parameters in units of frames. That is, the singing information of one song generated by each trained model is composed of multiple singing parameters (a group of singing parameters in time series) in units of frames. In this embodiment, one frame is defined as the length of one sample when the song is sampled at a predetermined sampling frequency (for example, 44.1 kHz) × 225.

[0028] The singing voice parameters for each frame include spectrum parameters (frequency spectrum of the voice to be pronounced) and fundamental frequency F0 parameters (pitch frequency of the voice to be pronounced). The spectrum parameters may be expressed as formant parameters, etc. The singing voice parameters may be expressed as filter coefficients, etc. In this embodiment, the filter coefficients to be applied for each frame are determined. Therefore, the present invention can be understood as a filter being changed for each frame.

[0029] Moreover, the frame-by-frame singing parameters include syllable information. FIG. 6 is an image diagram showing the relationship between frames and syllables. FIG. 6(a) is a diagram showing the relationship between frames and syllables in an English phrase, and FIG. 6(b) is a diagram showing the relationship between frames and syllables in a Japanese phrase. As shown in FIGS. 6(a) and 6(b), the voice of a song (phrase) is composed of a plurality of syllables (the first syllable (Come) and the second syllable (On) in FIG. 6(a), and the first syllable (Ka) and the second syllable (O) in FIG. 6(b). Each syllable is generally composed of one vowel, or a combination of one vowel and one or more consonants. That is, the singing voice parameters, which are parameters for pronouncing a syllable, include at least parameters corresponding to the vowels contained in the syllable. Each syllable is pronounced over a plurality of frame sections that are continuous in the time direction, and the syllable start position, syllable end position, vowel start position, and vowel end position (all of which are positions in the time direction) of each syllable contained in one song can be specified by the frame position (how many frames from the beginning). In the singing information, the singing parameters of the frames corresponding to the syllable start position, syllable end position, vowel start position, and vowel end position of each syllable include information such as the 0th syllable start frame, the 0th syllable end frame, the 0th vowel start frame, and the 0th vowel end frame (where 0 is a natural number).

[0030] Returning to Figure 5, in the electronic musical instrument 2, when singing voice information (first singing voice information generated by the trained model 302a and second singing voice information generated by the trained model 302b) is received from the terminal device 3 via the communication unit 208, the CPU 201 stores the received singing voice information in the RAM 203. Next, the CPU 201 sets singing voice information (singing voice parameter group) to be used for pronouncing the singing voice based on the operation information of the parameter change operator 103 input from the key scanner 206. Specifically, when the indicator 103a of the parameter change operator 103 is set to scale 1, the CPU 201 sets the first singing voice information as the parameter to be used for pronouncing the singing voice. When the indicator 103a of the parameter change operator 103 is set to scale 2, the CPU 201 sets the second singing voice information as the parameter to be used for pronouncing the singing voice. When the indicator 103a of the parameter change operator 103 is located between scale 1 and scale 2, the CPU 201 generates singing voice information based on the first singing voice information and the second singing voice information according to the position, stores it in the RAM 203, and sets the generated singing voice information as the parameter to be used for pronouncing the singing voice.

[0031] Next, the CPU 201 starts a singing voice sounding mode process (see FIG. 7) described later, detects the state of the keyboard 101 based on the performance operation information from the key scanner 206, and executes the voice synthesis processes A to D (see FIG. 8 to FIG. 11) to specify a frame to be sounded. Then, when the first mode is set, the CPU 201 reads out the fundamental frequency F0 parameter and the spectrum parameter of the specified frame of the set singing voice information from the RAM 203, and outputs them to the voice synthesis unit 205 together with the pitch information of the pressed key. The voice synthesis unit 205 generates singing voice waveform data based on the input pitch information, fundamental frequency F0 parameter, and spectrum parameter, and outputs them to the D / A converter 212. When the second mode is set, the CPU 201 reads out the spectrum parameter of the specified frame of the set singing voice information from the RAM 203, and outputs it to the voice synthesis unit 205. Also, the CPU 201 outputs the pitch information of the pressed key to the sound source unit 204. The sound source unit 204 reads out waveform data of a preset tone color corresponding to the input pitch information from the waveform ROM as vocal sound source waveform data, and outputs the data to the voice synthesis unit 205. The voice synthesis unit 205 generates singing voice waveform data based on the input vocal sound source waveform data and spectrum parameters, and outputs the generated data to the D / A converter 212. The singing voice waveform data output to the D / A converter 212 is converted into an analog audio signal, amplified by an amplifier 213 and output from a speaker 214 .

[0032] The singing voice pronunciation mode process will now be described. 7 is a flow chart showing the flow of the singing voice pronunciation mode process. The singing voice pronunciation mode process is executed by the CPU 201 in cooperation with a program stored in the ROM 202, for example, when the setting of singing voice information (singing voice parameter group) used for pronunciation of the singing voice is completed.

[0033] First, the CPU 201 initializes variables used in the voice synthesis processes A to D (step S1). Next, the CPU 201 determines, based on the input from the key scanner 206, whether or not an operation of the parameter change operator 103 has been detected (step S2). If it is determined that operation of the parameter change operator 103 has been detected (step S2; YES), the CPU 201 changes the singing voice information (group of singing voice parameters) used to pronounce the singing voice according to the position of the indication portion 103a of the parameter change operator 103 (step S3), and proceeds to step S4.

[0034] For example, when the indicator 103a of the parameter change operator 103 is changed to a state corresponding to the scale 1, the setting of the parameter used for the pronunciation of the singing voice is changed to the first singing voice information. When the indicator 103a of the parameter change operator 103 is changed to a state corresponding to the scale 2, the setting of the parameter used for the pronunciation of the singing voice is changed to the second singing voice information. When the indicator 103a of the parameter change operator 103 is changed to a state corresponding to a position between the scale 1 and the scale 2, singing voice information is generated based on the first singing voice information and the second singing voice information (for example, the first singing voice information and the second singing voice information are synthesized according to the ratio of the rotation angle from the scale 1 of the indicator 103a and the rotation angle from the scale 2), and stored in the RAM 203, and the setting of the parameter used for the pronunciation of the singing voice is changed to the generated singing voice information. This makes it possible to change the tone of the voice even during pronunciation (playing).

[0035] When it is determined that the operation of the parameter change operator 103 has not been detected (step S2; NO), the CPU 201 proceeds to step S4.

[0036] In step S4, the CPU 201 determines whether or not a key depression operation (KeyOn) of the keyboard 101 has been detected based on the performance operation information input from the key scanner 206 (step S4). If it is determined that KeyOn has been detected (step S4; YES), the CPU 201 executes a voice synthesis process A (step S5).

[0037] 8 is a flowchart showing the flow of the voice synthesis process A. The voice synthesis process A is executed by the CPU 201 in cooperation with a program stored in the ROM 202.

[0038] In the voice synthesis process A, first, the CPU 201 sets the KeyOnCounter to KeyOnCounter+1 (step S501). Here, KeyOnCounter is a variable that stores the number of keys that are currently being pressed (the number of operators that are currently being operated).

[0039] Next, the CPU 201 determines whether or not the KeyOnCounter is 1 (step S502). That is, it is determined whether the detected key depression is performed in a state where no other operating element is depressed.

[0040] If it is determined that KeyOnCounter is 1 (step S502; YES), the CPU 201 determines whether or not CurrentFramePos is the frame position of the last syllable (step S503). This CurrentFramePos is a variable that stores the frame position of the frame currently to be pronounced, and stores the frame position of the previously pronounced frame until it is replaced with the frame position of the next frame to be pronounced (for example, in FIG. 8, until step S508 or step S509 is executed).

[0041] If it is determined that CurrentFramePos is the frame position of the last syllable (step S503; YES), the CPU 201 sets the syllable start position of the first syllable to NextFramePos, which is a variable that stores the frame position of the next frame to be pronounced (step S504). Then, the CPU 201 sets NextFramePos to CurrentFramePos (step S509), and the process proceeds to step S510. That is, if the previously pronounced frame is the last syllable, there is no syllable following the previously pronounced syllable, so the position of the frame to be pronounced advances to the frame at the start position of the first syllable.

[0042] If it is determined that CurrentFramePos is not the frame position of the last syllable (step S503; NO), the CPU 201 sets the syllable start position of the next syllable in NextFramePos (step S505). Then, the CPU 201 sets NextFramePos to CurrentFramePos (step S509), and the process proceeds to step S510. That is, if the previously pronounced frame is not the last syllable, the position of the frame to be pronounced advances to the syllable start position of the next syllable.

[0043] On the other hand, if it is determined that KeyOnCounter is not 1 (step S502; NO), the CPU 201 sets NextFramePos to CurrentFramePos+playback rate / 120 (step S507). Here, 120 is the default tempo value, but the default tempo value is not limited to this. The playback rate is a value preset by the user. For example, when the playback rate is set to 240, the position of the next frame to be sounded is set to a position two frames ahead of the current frame position. When the playback rate is set to 60, the position of the next frame to be sounded is set to a position 0.5 frames ahead of the current frame position.

[0044] Next, the CPU 201 determines whether NextFramePos>vowel end position (step S507). That is, it determines whether the position of the frame to be pronounced next exceeds the vowel end position of the current syllable to be pronounced (i.e., the vowel end position of the previously pronounced syllable). If it is determined that NextFramePos is not greater than the vowel end position (step S507; NO), the CPU 201 sets NextFramePos to CurrentFramePos (step S509), and proceeds to step S510. That is, the frame position of the frame to be sounded advances to NextFramePos.

[0045] If it is determined that NextFramePos>vowel end position (step S507; YES), the CPU 201 sets the vowel end position of the current syllable to be pronounced to CurrentFramePos (step S508), and proceeds to step S510. That is, when NextFramePos exceeds the vowel end position, the frame position of the frame to be pronounced is not shifted to the position of NextFramePos, but is maintained at the vowel end position of the previously pronounced syllable.

[0046] In step S510, the CPU 201 retrieves from the RAM 203 the singing voice parameters of the frame at the frame position stored in CurrentFramePos of the singing voice information set as parameters used for pronouncing the singing voice, and outputs them to the voice synthesis unit 205 (step S510), and causes the voice synthesis unit 205 to generate singing voice waveform data based on the output singing voice parameters, and outputs the singing voice (voice) via the D / A converter 212, the amplifier 213, and the speaker 214 (step S511), and proceeds to step S6 in Figure 7.

[0047] Here, when the first mode is set, the CPU 201 outputs pitch information of the pressed key to the voice synthesis unit 205, reads out the fundamental frequency F0 parameter and the spectrum parameter of the specified frame of the set singing voice information from the RAM 203, and outputs them to the voice synthesis unit 205, causes the voice synthesis unit 205 to generate singing voice waveform data based on the output pitch information, fundamental frequency F0 parameter and spectrum parameter, and outputs (sounds) a voice based on the singing voice waveform data via the D / A converter 212, the amplifier 213, and the speaker 214. When the second mode is set, the CPU 201 reads out the spectrum parameter of the specified frame of the set singing voice information from the RAM 203, and outputs it to the voice synthesis unit 205. Also, the CPU 201 outputs pitch information of the pressed key to the sound source unit 204, and causes the sound source unit 204 to read out waveform data of a preset tone color according to the input pitch information from the waveform ROM as vocal sound source waveform data, and outputs it to the voice synthesis unit 205. Then, the voice synthesis unit 205 generates singing voice waveform data based on the input vocal sound source waveform data and spectrum parameters, and outputs a voice based on the singing voice waveform data via the D / A converter 212, the amplifier 213, and the speaker 214.

[0048] 7, the CPU 201 judges whether or not KeyOnCounter=1 (step S6). That is, it judges whether or not the currently detected key depression is a key depression in a state where no key is depressed. When it is determined that KeyOnCounter=1 (step S6; YES), the CPU 201 controls the amplifier 213 to perform pronunciation start processing (fade-in) of the voice based on the generated singing voice waveform data (step S7), and proceeds to step S17. The pronunciation start processing is a processing for gradually increasing (fading in) the volume of the amplifier 213 until it reaches a set value. This allows the voice based on the singing voice waveform data generated by the voice synthesis unit 205 to be output (sounded) from the speaker 214 while gradually increasing the volume. Note that when the volume of the amplifier 213 reaches the set value, the pronunciation start processing ends, but the volume of the amplifier 213 is maintained at the set value until the mute start processing is executed. When it is determined that KeyOnCounter=1 is not satisfied (step S6; NO), the CPU 201 proceeds to step S17. In other words, if there is a key that has already been pressed at the time of the currently detected key pressing operation, the sound generation start process has already started, so the CPU 201 proceeds directly to step S17.

[0049] On the other hand, if it is determined in step S4 that KeyOn has not been detected (step S4; NO), the CPU 201 determines whether or not a key release (KeyOff, i.e., release of a key depression operation) of any key on the keyboard 101 has been detected (step S8).

[0050] In step S8, if it is determined that KeyOff has not been detected (step S8; NO), the CPU 201 determines whether or not KeyOnCounter=>1 (step S9). If it is determined that KeyOnCounter=>1 (step S9; YES), the CPU 201 executes a voice synthesis process B (step S10).

[0051] 9 is a flowchart showing the flow of the voice synthesis process B. The voice synthesis process B is executed by the CPU 201 in cooperation with a program stored in the ROM 202. In the voice synthesis process B, first, the CPU 201 sets NextFramePos to CurrentFramePos+playback rate / 120 (step S901). The process of step S901 is similar to that of step S506 in FIG. 8, and therefore the description thereof will be cited.

[0052] Next, the CPU 201 determines whether NextFramePos>vowel end position (step S902). That is, it determines whether NextFramePos exceeds the vowel end position of the current syllable to be pronounced (i.e., the vowel end position of the previously pronounced syllable). If it is determined that NextFramePos is not greater than the vowel end position (step S902; NO), the CPU 201 sets NextFramePos to CurrentFramePos (step S903), and the process proceeds to step S905. That is, if NextFramePos does not exceed the vowel end position, the frame position of the frame to be pronounced is advanced to NextFramePos.

[0053] If it is determined that NextFramePos>vowel end position (step S902; YES), the CPU 201 sets the vowel end position of the current syllable to be pronounced to CurrentFramePos (step S904), and proceeds to step S905. That is, when NextFramePos exceeds the vowel end position, the frame position of the frame to be pronounced is not shifted to the position of NextFramePos, but is maintained at the vowel end position of the previously pronounced syllable.

[0054] In step S905, the CPU 201 retrieves from the RAM 203 the singing voice parameters of the frame at the frame position stored in CurrentFramePos of the singing voice information set as parameters used for pronouncing the singing voice, and outputs them to the voice synthesis unit 205 (step S905), and causes the voice synthesis unit 205 to generate singing voice waveform data based on the output singing voice parameters, and outputs the singing voice via the D / A converter 212, the amplifier 213, and the speaker 214 (step S906), and proceeds to step S17 in Figure 7. The processes in steps S905 and S906 are similar to those in steps S510 and S511 in FIG. 8, respectively, and therefore the explanation thereof will be cited.

[0055] On the other hand, when it is determined in step S8 of FIG. 7 that KeyOff has been detected (step S8; YES), the CPU 201 executes a voice synthesis process C (step S11).

[0056] 10 is a flowchart showing the flow of the voice synthesis process C. The voice synthesis process C is executed by the CPU 201 in cooperation with a program stored in the ROM 202.

[0057] In the voice synthesis process C, first, the CPU 201 sets KeyOnCounter to KeyOnCounter-1 (step S1101). Next, the CPU 201 sets NextFramePos to CurrentFramePos+playback rate / 120 (step S1102). The process of step S1102 is similar to that of step S506 in FIG. 8, and therefore the description thereof will be cited.

[0058] Next, the CPU 201 determines whether NextFramePos>vowel end position (step S1103). That is, it determines whether NextFramePos exceeds the vowel end position of the current syllable to be pronounced (i.e., the vowel end position of the previously pronounced syllable). If it is determined that NextFramePos is not greater than the vowel end position (step S1103; NO), the CPU 201 sets NextFramePos to CurrentFramePos (step S1107), and proceeds to step S1109. That is, if NextFramePos does not exceed the vowel end position, the frame position of the frame to be pronounced is advanced to NextFramePos.

[0059] If it is determined that NextFramePos>vowel end position (step S1103; YES), the CPU 201 determines whether or not KeyOnCounter=0 (that is, whether or not all keys of the keyboard 101 are released) (step S1104). If it is determined that KeyOnCounter is not 0 (step S1104; NO), the CPU 201 sets the vowel end position of the current syllable to be pronounced in CurrentFramePos (step S1105), and proceeds to step S1109. In other words, when NextFramePos exceeds the vowel end position and all keys on the keyboard 101 are not released (any keys are pressed), the frame position of the frame to be pronounced is not shifted to NextFramePos, but is maintained at the vowel end position of the previously pronounced syllable.

[0060] If it is determined that KeyOnCounter=0 (step S1104; YES), the CPU 201 determines whether NextFramePos>the syllable end position (step S1106). That is, the CPU 201 determines whether or not NextFramePos exceeds the syllable end position of the current syllable to be pronounced (that is, the syllable end position of the previously pronounced syllable).

[0061] If it is determined that NextFramePos is not greater than the syllable end position (step S1106; NO), the CPU 201 sets NextFramePos to CurrentFramePos (step S1107), and proceeds to step S1109. That is, when all the keys of the keyboard 101 are released and NextFramePos does not exceed the syllable end position, the frame position of the frame to be sounded is advanced to NextFramePos.

[0062] If it is determined that NextFramePos>the syllable end position (step S1106; YES), the CPU 201 sets the syllable end position to CurrentFramePos (step S1108), and proceeds to step S1109. In other words, when all the keys on the keyboard 101 are released and NextFramePos exceeds the syllable end position, the frame position of the frame to be sounded is not shifted to NextFramePos, but is maintained at the syllable end position of the previously sounded syllable.

[0063] In step S1109, the CPU 201 retrieves from the RAM 203 the singing voice parameters for the frame at the frame position stored in CurrentFramePos of the singing voice information, which is set as the parameters used for pronouncing the singing voice, and outputs them to the voice synthesis unit 205 (step S1109), and causes the voice synthesis unit 205 to generate singing voice waveform data based on the output singing voice parameters, and outputs the singing voice via the D / A converter 212, the amplifier 213, and the speaker 214 (step S1110), and proceeds to step S12 in Figure 7. The processes in steps S1109 and S1110 are similar to those in steps S510 and S511 in FIG. 8, respectively, and therefore the same explanation will be used.

[0064] In step S12 of FIG. 7, the CPU 201 determines whether or not KeyOnCounter=0 (whether or not all keys on the keyboard 101 have been released) (step S12). If it is determined that KeyOnCounter=0 is not met (no key release has been detected for any of the keys on the keyboard 101) (step S12; NO), the CPU 201 proceeds to step S17. If it is determined that KeyOnCounter=0 (that is, all keys on the keyboard 101 have been released) (step S12; YES), the CPU 201 controls the amplifier 213 to execute a mute start process (start fade-out) (step S13), and proceeds to step S17. The mute start process is a process for starting a mute process in which the volume of the amplifier 213 is gradually reduced until it becomes 0. By the mute process, the voice based on the singing voice waveform data generated by the voice synthesis unit 205 is output from the speaker 214 at a gradually decreasing volume.

[0065] On the other hand, if it is determined in step S9 that KeyOnCounter>=1 is not satisfied (step S9; NO), that is, if it is determined that all keys on the keyboard 101 are released, the CPU 201 determines whether the volume of the amplifier 213 is 0 or not (step S14). When it is determined that the volume of the amplifier 213 is not 0 (step S14; NO), the CPU 201 executes a voice synthesis process D (step S15).

[0066] 11 is a flowchart showing the flow of the voice synthesis process D. The voice synthesis process D is executed by the CPU 201 in cooperation with a program stored in the ROM 202.

[0067] In the voice synthesis process D, first, the CPU 201 sets NextFramePos to CurrentFramePos+playback rate / 120 (step S1501). The process of step S1501 is similar to that of step S506 in FIG. 8, and therefore the description thereof will be cited.

[0068] Next, the CPU 201 determines whether NextFramePos>vowel end position (step S1502). That is, it determines whether NextFramePos exceeds the vowel end position of the current syllable to be pronounced (i.e., the vowel end position of the previously pronounced syllable). If it is determined that NextFramePos is not greater than the vowel end position (step S1502; NO), the CPU 201 sets NextFramePos to CurrentFramePos (step S1504), and the process proceeds to step S1506. That is, if NextFramePos does not exceed the vowel end position, the frame position of the frame to be pronounced is advanced to NextFramePos.

[0069] If it is determined that NextFramePos>vowel end position (step S1502; YES), the CPU 201 determines whether NextFramePos>syllable end position (step S1503). That is, the CPU 201 determines whether or not NextFramePos exceeds the syllable end position of the current syllable to be pronounced (that is, the syllable end position of the previously pronounced syllable).

[0070] If it is determined that NextFramePos is not greater than the syllable end position (step S1503; NO), the CPU 201 sets NextFramePos to CurrentFramePos (step S1504) and proceeds to step S1506. In other words, if NextFramePos does not exceed the syllable end position, the frame position of the frame to be pronounced is advanced to NextFramePos.

[0071] If it is determined that NextFramePos>the syllable end position (step S1503; YES), the CPU 201 sets the syllable end position to CurrentFramePos (step S1505), and proceeds to step S1506. That is, when NextFramePos exceeds the syllable end position, the frame position of the frame to be pronounced is not shifted to NextFramePos, but is maintained at the syllable end position of the previously pronounced syllable.

[0072] In step S1506, the CPU 201 retrieves from the RAM 203 the singing voice parameters of the frame at the frame position stored in CurrentFramePos of the singing voice information set as parameters to be used for pronouncing the singing voice, and outputs them to the voice synthesis unit 205 (step S1506), causes the voice synthesis unit 205 to generate singing voice waveform data based on the output singing voice parameters, and outputs the singing voice via the D / A converter 212, the amplifier 213, and the speaker 214 (step S1507), and proceeds to step S16 in Figure 7. The processes in steps S1506 and S1507 are similar to those in steps S510 and S511 in FIG. 8, respectively, and therefore the explanation thereof will be cited.

[0073] In step S16 of FIG. 7, the CPU 201 controls the amplifier 213 to execute a sound muting process (fade-out) (step S16), and the process proceeds to step S17.

[0074] On the other hand, in step S14, when it is determined that the volume of the amplifier 213 is 0 (step S14; YES), the CPU 201 proceeds to step S17.

[0075] In step S17, the CPU 201 determines whether or not an instruction to end the singing voice pronunciation mode has been issued (step S17). For example, when the singing voice pronunciation mode switch is pressed to instruct a transition to the normal mode, the CPU 201 determines that an instruction to end the singing voice pronunciation mode has been given.

[0076] When it is determined that the end of the singing voice pronunciation mode has not been instructed (step S17; NO), the CPU 201 returns to step S2. When it is determined that an instruction to end the singing voice pronunciation mode has been given (step S17; YES), the CPU 201 ends the singing voice pronunciation mode process.

[0077] FIG. 12 shows a graph indicating the change in volume from the time of key depression detection (when no key is depressed) to the time of key release (KeyOff) when the volume becomes 0, when the syllable Come is generated in response to the operation of the keyboard 101 (key depression (KeyOn)) in the singing voice sound generation mode process described above, and a diagram showing frame positions used for sound generation at each timing of the graph. FIG. 12(a) shows a graph and a diagram showing a case where key release (all keys released) is detected at the timing of the end position of the vowel ah. FIG. 12(b) shows a graph and a diagram showing a case where key release (all keys released) is detected three frames after the timing of the end position of the vowel ah. FIG. 12(c) shows a case where key release (all keys released) is detected before the end position of the vowel ah.

[0078] As shown in Fig. 12(b), after starting the pronunciation of a syllable based on the singing parameters of a syllable start frame (the first frame in Fig. 12(b)) by key pressing detection, if the key continues to be pressed even after the frame position advances to the frame (a certain vowel frame) of the vowel end position in the vowel section (the ah section in Fig. 12(b)) included in the sounded syllable (i.e., after the start of the pronunciation of the vowel based on the singing parameters of the frame of the vowel end position), the pronunciation of the vowel continues based on the singing parameters of the frame of the vowel end position until the key release (all keys released) is detected. Also, as shown in Fig. 12(c), after starting the pronunciation of a syllable based on the singing parameters of a syllable start frame (the first frame in Fig. 12(c)) by key pressing detection, if the key release (all keys released) is detected before the frame position advances to the vowel end position, the sound muffling process is immediately started, and the sound muffling process is performed while advancing the frame position used for the singing parameters. Therefore, syllables can be naturally pronounced with lengths that correspond to the operation of the keyboard 101 by the user.

[0079] In a conventional singing voice pronunciation technique using an electronic musical instrument (for example, Patent Document 1), audio information, which is waveform data of a plurality of pronunciation units, is connected to perform pronunciation for each syllable or loop playback according to an operation, so that it is difficult to produce a natural singing voice. In addition, since it is necessary to store audio information in which the waveform data of each of a plurality of vocalization units is time-series, a large memory capacity is required. In the electronic musical instrument 2 of this embodiment, if a key is continuously pressed even after the start of the pronunciation of a vowel based on the frame at the vowel end position of a syllable, singing voice waveform data is generated and produced using the singing voice parameters of the frame at the vowel end position among the singing voice parameters generated by a trained model that has trained human singing voices by machine learning, so that a more natural voice (singing voice) can be produced, instead of an awkward pronunciation as in the case where vowel waveforms are connected. In addition, since it is not necessary to store the waveform data of each of a plurality of vocalization units in the RAM 203, a smaller memory capacity is required than in the conventional singing voice pronunciation technique.

[0080] Furthermore, conventional singing voice pronunciation techniques using electronic musical instruments reproduce waveform data, so the pronunciation is fixed and the tone cannot be changed during reproduction. On the other hand, the electronic musical instrument 2 of this embodiment uses singing voice parameters to generate a voice waveform and perform pronunciation, so it is possible to change the tone of the singing voice in response to the user's operation of the parameter change operator 103 while the singing voice is being pronounced (played).

[0081] As described above, according to the CPU 201 of the electronic musical instrument 2, after starting to produce a syllable based on parameters corresponding to a syllable start frame in response to detection of a key depression operation of the keyboard 101, if the state in which the pressed key remains present even after starting to produce a vowel based on parameters corresponding to a certain vowel frame within a vowel section contained in the syllable, the CPU 201 continues to produce the vowel based on parameters corresponding to the certain vowel frame until the key depression is released (i.e., until a key release is detected). Specifically, it outputs singing voice parameters corresponding to a certain vowel frame to the voice synthesis unit 205 of the electronic musical instrument 2, causes the voice synthesis unit 205 to generate voice waveform data based on the singing voice parameters, and produces a voice based on the voice waveform data. Therefore, it is possible to produce more natural sounds in response to the operation of the electronic musical instrument with a smaller memory capacity.

[0082] In addition, the singing parameters used to pronounce syllables are inferred from a trained model generated by machine learning of the human (singer) voice, making it possible to produce expressive pronunciation that retains the singer's natural phoneme-level pronunciation nuances.

[0083] Furthermore, the CPU 201 changes the vocal parameters for pronouncing syllables to vocal parameters of a different tone in response to the operation of the parameter change operator 103 executed by the user at any timing including during performance. Therefore, it is possible to change the tone of the vocal even during performance (pronunciation of vocal).

[0084] It should be noted that the contents described in the above embodiment are preferred examples of the information processing device, electronic musical instrument, electronic musical instrument system, method, and program according to the present invention, and are not intended to be limiting. For example, in the above embodiment, the information processing device of the present invention has been described as being included in the electronic musical instrument 2, but the present invention is not limited to this. For example, the functions of the information processing device of the present invention may be implemented by an external device (such as the above-mentioned terminal device 3 (PC (Personal Computer), tablet terminal, smartphone, etc.) connected to the electronic musical instrument 2 via a wired or wireless communication interface. )) may be provided.

[0085] In the above embodiment, the trained model 302a and the trained model 302b are described as being provided in the terminal device 3, but they may be provided in the electronic musical instrument 2. The trained model 302a and the trained model 302b may infer singing voice information based on lyric data and pitch data input in the electronic musical instrument 2.

[0086] In the above embodiment, the syllable is generated when a key depression operation on one key is detected while none of the keys on the keyboard 101 is being depressed, but the key depression operation that triggers the generation of a syllable is not limited to this. For example, the syllable may be generated when a key depression operation on a melody line (top note) is detected.

[0087] In the above embodiment, the electronic musical instrument 2 is an electronic keyboard instrument, but the electronic musical instrument is not limited to this and may be other electronic musical instruments such as an electronic string instrument or an electronic wind instrument.

[0088] In the above embodiment, an example is disclosed in which a semiconductor memory such as a ROM or a hard disk is used as a computer-readable medium for the program according to the present invention, but the present invention is not limited to this example. As other computer-readable media, a portable recording medium such as an SSD or a CD-ROM can be applied. In addition, a carrier wave can be applied as a medium for providing data of the program according to the present invention via a communication line.

[0089] In addition, the detailed configurations and operations of the electronic musical instrument, information processing device, and electronic musical instrument system may be modified as appropriate without departing from the spirit and scope of the invention.

[0090] Although the embodiment of the present invention has been described above, the technical scope of the present invention is not limited to the above embodiment, but is defined based on the claims. Furthermore, the technical scope of the present invention also includes equivalent ranges that are made by adding modifications that are not related to the essence of the present invention from the claims. The inventions described in the claims originally attached to this application are set forth below. The claim numbers in the appended claims are the same as those in the claims originally attached to this application. [Additional Notes] <Claim 1> a control unit which starts pronunciation of a syllable based on parameters corresponding to a syllable start frame in response to detection of an operation on an operator, and then, when the operation on the operator continues even after the start of pronunciation of a vowel based on parameters corresponding to a certain vowel frame in a vowel section included in the syllable, continues pronouncing the vowel based on the parameters corresponding to the certain vowel frame until the operation on the operator is released; An information processing device comprising: <Claim 2> 2. The information processing device according to claim 1, wherein the control unit outputs the parameters to a voice synthesis unit of an electronic musical instrument, causes the voice synthesis unit to generate voice waveform data based on the parameters, and produces a voice based on the voice waveform data. <Claim 3> The information processing device according to claim 1 or 2, wherein the parameters are parameters inferred by a trained model generated by machine learning of human voices. <Claim 4> 4. The information processing device according to claim 1, wherein the parameters include spectral parameters. <Claim 5> 5. The information processing device according to claim 1, wherein the control unit changes the parameter to a parameter of another timbre in response to an instruction operation to change the timbre of the voice to be produced, performed by a user at any timing including during performance. <Claim 6> The case where the operation of the operator is continued includes the case where a key is being pressed in an electronic keyboard instrument, 6. The information processing device according to claim 1, wherein the release of the operation on the operator includes a state in which all pressed keys in the electronic keyboard instrument are released and no keys are pressed. <Claim 7> An information processing device according to any one of claims 1 to 6; A plurality of operators; An electronic musical instrument comprising: <Claim 8> An information processing device according to any one of claims 1 to 6; an electronic musical instrument having a plurality of operators; An electronic musical instrument system comprising: <Claim 9> A control unit of the information processing device A method for starting pronunciation of a syllable based on parameters corresponding to a syllable start frame in response to detection of an operation on an operator, and then, if the operation on the operator continues even after the start of pronunciation of a vowel based on parameters corresponding to a certain vowel frame within a vowel section contained in the syllable, continuing to pronounce the vowel based on the parameters corresponding to the certain vowel frame until the operation on the operator is released. <Claim 10> A control unit of the information processing device after starting pronunciation of a syllable based on parameters corresponding to a syllable start frame in response to detection of an operation on an operator, if the operation on the operator continues even after the start of pronunciation of a vowel based on parameters corresponding to a certain vowel frame in a vowel section included in the syllable, continuing pronunciation of the vowel based on parameters corresponding to the certain vowel frame until the operation on the operator is released; A program for executing a process. [Explanation of symbols]

[0091] 1. Electronic Musical Instrument System 2. Electronic Instruments 101 Keyboard 102 Switch Panel 103 Parameter change controls 104 LCD 201 CPU 202 ROM 203 RAM 204 Sound Source Section 205 Voice synthesis unit 206 Key Scanner 208 Communications Department 209 Bus 210 Timer 211 D / A Converter 212 D / A Converter 213 Amplifier 214 Speaker 3 Terminal Equipment 301 CPU 302 ROM 302a Pre-trained model 302b Pre-trained model 303 RAM 304 Storage section 305 Operation section 306 Display section 307 Communications Department 308 Bus

Claims

1. when an operation on the operator is released while a vowel is being produced based on a vowel frame in response to an operation on the operator, it is determined whether or not a consonant frame follows the vowel frame; if the consonant frame is present, a pronunciation process of the consonant based on the consonant frame is executed, and a mute process is executed for the pronunciation process of the consonant. An information processing device having a control unit.

2. The control unit is 2. The information processing apparatus according to claim 1, further comprising: an information processing unit that outputs parameters to a voice synthesis unit of an electronic musical instrument, causes the voice synthesis unit to generate voice waveform data based on the parameters, and produces a voice based on the voice waveform data.

3. The information processing device according to claim 2 , wherein the parameters are parameters inferred by a trained model generated by machine learning of human voices.

4. The information processing apparatus according to claim 2 , wherein the parameters include spectral parameters.

5. The information processing device according to any one of claims 2 to 4, wherein the control unit changes the parameter to a parameter of another tone color in response to an instruction operation to change the tone color of the sound to be produced, performed by a user at any timing including during performance.

6. An information processing device as described in any one of claims 1 to 5, wherein release of the operation on the operator includes a state in which, in an electronic keyboard instrument, all pressed keys are released and no keys are pressed.

7. An information processing device according to any one of claims 1 to 6; A plurality of operators; An electronic musical instrument comprising:

8. An information processing device according to any one of claims 1 to 6; an electronic musical instrument having a plurality of operators; An electronic musical instrument system comprising:

9. A control unit of the information processing device when an operation on the operator is released while a vowel is being produced based on a vowel frame in response to an operation on the operator, it is determined whether or not a consonant frame follows the vowel frame; if the consonant frame is present, a pronunciation process of the consonant based on the consonant frame is executed, and a mute process is executed for the pronunciation process of the consonant. method.

10. A control unit of the information processing device when an operation on the operator is released while a vowel is being produced based on a vowel frame in response to an operation on the operator, it is determined whether or not a consonant frame follows the vowel frame; if the consonant frame is present, a pronunciation process of the consonant based on the consonant frame is executed, and a mute process is executed for the pronunciation process of the consonant. Program for.

Images