Sound control device and control method thereof, electronic musical instrument, and program

Abstract

Provided is a sound control device for: acquiring musical performance information that includes pitch information indicating a pitch and volume information indicating a volume detected as a continuous quantity; determining a start of note and an end of note on the basis of a result of comparison between a threshold value and the volume in the musical performance information and a change in the pitch in the musical performance information; and, if it is determined that the volume has changed in a predetermined manner, determining the end of note and the start of note at a timing when the volume has changed in the predetermined manner, regardless of the result of comparison between the volume and the threshold value.

Description

[Technical field]

[0001] The present invention relates to a sound control device, a control method thereof, an electronic musical instrument, and a program. [Background technology]

[0002] There are known sound control devices for musical instruments that determine the start and end of notes in real time according to playing operations. For example, Patent Documents 1 and 2 disclose techniques for determining notes according to playing operations and generating synthetic singing sounds in real time. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-206496 A [Patent Document 2] JP 2014-98801 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, depending on the type of electronic musical instrument, when determining notes based on performance information generated by a performance operation, it is not always easy to determine the start and end of a note. Therefore, there is room for improvement in determining the start and end of a note in accordance with the user's intention.

[0005] An object of the present invention is to provide a sound control device that can determine the start and end of notes in accordance with the performer's intention. [Means for solving the problem]

[0006] In order to achieve the above object, the present invention includes an acquisition unit that acquires performance information including pitch information indicating a pitch and volume information indicating a volume detected as a continuous quantity, and a determination unit that determines a note start and a note end based on a comparison result between a volume in the performance information and a threshold value and a change in the pitch in the performance information, and when the determination unit determines that the volume has changed in a predetermined manner, the determination unit determines that the note has ended and that the note has started at the timing when the volume has changed in the predetermined manner, regardless of the comparison result between the volume and the threshold value. and the predetermined aspect is that the volume drops at a speed faster than the first predetermined speed within a first predetermined time, and then, before a second predetermined time has elapsed from the start of the drop, the volume continues to rise at a speed faster than the second predetermined speed and for a third predetermined time from the start of the rise. A sound control device is provided. Effect of the Invention

[0007] According to the present invention, it is possible to determine the start and end of notes in accordance with the performer's intention. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram of a sound control system including a sound control device. [Diagram 2] FIG. 4 is a diagram showing lyrics data. [Diagram 3] FIG. 2 is a functional block diagram of the sound control device. [Figure 4] FIG. 13 is a diagram illustrating a method for identifying a syllable. [Diagram 5] FIG. 13 is a diagram illustrating a method for identifying a syllable. [Figure 6] FIG. 13 is a diagram illustrating a method for identifying a syllable. [Figure 7] 13 is a flowchart showing a sound generation process. [Figure 8] FIG. 2 is a diagram showing the relationship between musical notation and notes to which syllables are assigned. [Figure 9] 13A and 13B are diagrams illustrating an example of generating syllable notes and examples of countermeasures. [Figure 10] FIG. 11 is a diagram showing an example of a change in acquired volume. [Figure 11] 13 is a flowchart showing an instruction process. [Figure 12] 13 is a flowchart showing an instruction process. [Figure 13] 13 is a flowchart showing an instruction process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0010] (First embodiment) 1 is a block diagram of a sound control system including a sound control device according to a first embodiment of the present invention. The sound control system includes a sound control device 100 and an external device 20. The sound control device 100 is, for example, an electronic musical instrument, and may be, for example, an electronic wind instrument in the form of a saxophone or the like.

[0011] The sound control device 100 includes a control unit 11, an operation unit 12, a display unit 13, a storage unit 14, a performance operation unit 15, a sound generation unit 18, and a communication I / F (interface) 19. These elements are connected to each other via a communication bus 10.

[0012] The control unit 11 includes a CPU 11a, a ROM 11b, a RAM 11c, and a timer (not shown). A control program executed by the CPU 11a is stored in the ROM 11b. The CPU 11a implements various functions of the sound control device 100 by expanding the control program stored in the ROM 11b into the RAM 11c and executing it. The various functions here include, for example, a function for executing a sound generation process. The sound generation function will be described later.

[0013] The control unit 11 includes a DSP (Digital Signal Processor) for generating an audio signal by a sound generation function. The storage unit 14 is a non-volatile memory. The storage unit 14 stores setting information used when generating an audio signal indicating a synthetic singing voice, as well as voice segments for generating a synthetic singing voice. The setting information includes, for example, a tone color and acquired lyric data. Note that some or all of this information and data may be stored in the storage unit 14.

[0014] The operation unit 12 includes a plurality of operators for inputting various information and receives instructions from a user. The display unit 13 displays various information. The sound generation unit 18 includes a sound source circuit, an effect circuit, and a sound system.

[0015] The performance operation unit 15 includes a plurality of operation keys 16 and a breath sensor 17 as elements for inputting a performance signal (performance information). The input performance signal includes pitch information indicating the pitch and volume information indicating the volume detected as a continuous quantity, and is supplied to the control unit 11. The main body of the sound control device 100 is provided with a plurality of tone holes (not shown). When a user (performer) plays the plurality of operation keys 16, the open / closed state of the tone holes changes, and a desired pitch is specified.

[0016] A mouthpiece is attached to the main body of the sound control device 100, and the breath sensor 17 is provided near the mouthpiece. The breath sensor 17 is a blowing pressure sensor that detects the pressure of the breath blown by the user through the mouthpiece. The breath sensor 17 detects the presence or absence of breath, and detects the strength and speed (force) of the blowing pressure during performance. The volume is specified according to the change in pressure detected by the breath sensor 17. The magnitude of the pressure that changes over time, detected by the breath sensor 17, is treated as volume information detected as a continuous quantity.

[0017] The communication I / F 19 is connected to the communication network wirelessly or by wire. For example, the sound control device 100 is connected to an external device 20 via the communication network by the communication I / F 19 so as to be able to communicate with the external device 20. The communication network may be, for example, the Internet, and the external device 20 may be a server device. The communication network may be a short-range wireless communication network using Bluetooth (registered trademark), infrared communication, LAN, or the like. The number and types of external devices connected are not important. The communication I / F 19 may include a MIDI I / F that transmits and receives MIDI (Musical Instrument Digital Interface) signals.

[0018] The external device 20 stores music data necessary for providing karaoke in association with a music ID. The music data includes data related to the song to be karaoke, such as lead vocal data, chorus data, accompaniment data, and karaoke subtitle data. The accompaniment data is data indicating the accompaniment sound of the song to be sung. The lead vocal data, chorus data, and accompaniment data may be data expressed in MIDI format. The karaoke subtitle data is data for displaying lyrics on the display unit 13.

[0019] The external device 20 also stores setting data in association with the song ID. This setting data is data that is set in the sound control device 100 according to the song to realize the synthesis of singing sounds. The setting data includes lyric data corresponding to each part of the song corresponding to the song ID. For example, this lyric data is lyric data corresponding to the lead vocal part. The song data and the setting data are associated with each other in terms of time.

[0020] This lyrics data may be the same as the karaoke subtitle data, or may be different. In other words, the lyrics data is the same in that it is data that specifies the lyrics (characters) to be spoken, but it is adjusted to a format that is easy to use in the sound control device 100.

[0021] For example, the karaoke subtitle data is a character string of "ko", "n", "ni", "chi", and "ha". In contrast, the lyrics data may be a character string that matches the actual pronunciation of "ko", "n", "ni", "chi", and "wa" so that it can be easily used by the sound control device 100. In addition, this format may include, for example, information that identifies when two characters are sung with one sound, information that identifies the division of phrases, and the like.

[0022] In the sound generation process, the control unit 11 acquires the music data and setting data designated by the user from the external device 20 via the communication I / F 19, and stores them in the storage unit 14. As described above, the music data includes accompaniment data, and the setting data includes lyric data. Moreover, the accompaniment data and the lyric data are associated with each other in terms of time.

[0023] Fig. 2 is a diagram showing lyric data stored in the storage unit 14. Hereinafter, each lyric (character) to be uttered, that is, one phonetic unit (a group of sounds), may be referred to as a "syllable". As described in Fig. 2, in the lyric data used in this embodiment, a plurality of syllables to be pronounced are arranged in time series. Each syllable is associated with a set period (set section) consisting of a start time and a stop time of pronunciation.

[0024] The lyrics data is data that specifies the syllables to be uttered. The lyrics data has text data in which multiple syllables to be uttered are arranged in chronological order. The lyrics data includes timing data that specifies the start and stop times of utterance for each syllable on a predetermined time axis. The start and stop times are defined, for example, as times based on the beginning of the song. This timing data associates a progression position in the song with lyrics to be uttered at that progression position. Thus, the lyrics data is data in which multiple syllables to be pronounced are arranged in chronological order and a syllable corresponding to a time elapsed from a reference time is uniquely specified.

[0025] As shown in Fig. 2, the lyrics data includes text data indicating "ko", "n", "ni", "chi", "wa", "dra", "gon", "night", "dra", "gon", etc. The syllables indicating "ko", "n", "ni", "chi", "wa", "dra", "gon", "night", "dra", and "gon" correspond to M(i), and the order of the syllables in the lyrics is determined by "i" (i = 1 to n). For example, M(5) corresponds to the fifth syllable in the lyrics.

[0026] The lyrics data includes timing data in which a vocalization start time ts(i) and a vocalization stop time te(i) are set for each syllable M(i). For example, in the case of M(1) "ko", the vocalization start time is time ts(1) and the vocalization stop time is time te(1). Similarly, in the case of M(n) "ru", the vocalization start time is time ts(n) and the vocalization stop time is time te(n). The period from time ts(i) to time te(i) corresponding to each syllable M(i) is called the vocalization setting period of the syllable M(i). This vocalization setting period indicates, for example, the period in the case of ideal singing. As described below, the vocalization period of each syllable included in the synthetic singing sound is controlled based on a vocalization start instruction and a vocalization stop instruction by a performance signal.

[0027] 3 is a functional block diagram of the sound control device 100 for implementing the sound generation process. The sound control device 100 includes, as functional units, an acquisition unit 31, a determination unit 32, a generation unit 33, a specification unit 34, a singing sound synthesis unit 35, and an instruction unit 36. The functions of these functional units are realized by cooperation of the CPU 11a, ROM 11b, RAM 11c, a timer, a communication I / F 19, etc. Note that it is not essential to include the generation unit 33, the specification unit 34, the singing sound synthesis unit 35, and the instruction unit 36.

[0028] The acquisition unit 31 acquires a performance signal. The determination unit 32 determines the start and end of a note based on a comparison result between the volume of the performance signal and a threshold value and a change in pitch in the performance signal. The generation unit 33 generates a note based on the determination of the note start and note end. The identification unit 34 identifies a syllable corresponding to the timing determined as the note start from the lyric data. The method of identifying a syllable will be described with reference to Figs. 4 to 6.

[0029] The singing sound synthesis unit 35 synthesizes the specified syllables based on the setting data to generate a singing sound. The instruction unit 36 ​​instructs the singing sound of the specified syllable to start to be pronounced at a pitch and timing corresponding to the start of the note, and to end to be pronounced at a timing corresponding to the end of the note. Based on the instruction by the instruction unit 36, the singing sound obtained by synthesizing the syllables is pronounced by the pronunciation unit 18 (FIG. 1).

[0030] Next, an aspect of the sound generation process will be outlined. Lyric data and accompaniment data corresponding to a piece of music designated by the user are stored in the storage unit 14. When the user issues an instruction to start playing using the operation unit 12, playback of the accompaniment data begins. At that time, the lyrics in the lyric data (or the karaoke subtitle data) are displayed on the display unit 13 in accordance with the progression of the accompaniment data. Also, the musical score of the main melody corresponding to the lead vocal data is displayed on the display unit 13 in accordance with the progression of the accompaniment data. The user plays using the performance operation unit 15 while listening to the accompaniment data. As the performance progresses, the acquisition unit 31 acquires a performance signal.

[0031] The determination unit 32 determines that a note has started when the volume of the performance signal exceeds a first threshold TH1 (see FIG. 10), and determines that a note has ended when the volume falls below a second threshold TH2 (see FIG. 10) after determining that a note has started. The note start here corresponds to an instruction to start vocalization, and the note end here corresponds to an instruction to end vocalization. The first threshold TH1 is higher than the second threshold TH2 or is the same as the second threshold TH2.

[0032] Furthermore, when the pitch changes while the volume exceeds the first threshold value TH1, the determination unit 32 determines that the note has ended and that the note has started. In other words, when the pitch is changed by operating the operation key 16 with a finger while maintaining a certain level of blowing pressure, the note end of the pitch before the change and the note start of the pitch after the change are determined to occur at the same time.

[0033] The specified syllables are synthesized and pronounced as singing sounds at the timing determined as the start of the note, and are muted at the timing determined as the end of the note. Thus, the user can pronounce lyrics that match the music by playing along with the accompaniment sounds.

[0034] 4 to 6 are diagrams for explaining a method for identifying syllables, and show the relationship between time and notes.

[0035] The control unit 11 (identification unit 34) identifies a syllable corresponding to a set period to which the timing determined to be a note start belongs as the syllable corresponding to the timing determined to be a note start. In addition, when the timing determined to be a note start does not belong to any set period, the control unit 11 (identification unit 34) identifies a syllable corresponding to a set period closest to the timing as the syllable corresponding to the timing determined to be a note start.

[0036] First, a specific example will be described with reference to Fig. 4 for the case where the count value tc at which the vocalization start instruction (note start) is acquired is within the vocalization setting period ts(1) to te(1). The count value tc is a count value for progressing the accompaniment position in the sound generation process (Fig. 7) described later.

[0037] Assume that a performance signal including a vocalization start instruction associated with the pitch "G4" is received in a standby state in the sound generation process. In this case, the control unit 11 sets the pitch "G4" and, with reference to the lyrics data, determines whether the count value tc at which the vocalization start instruction was acquired is included in the vocalization setting period. Because the time at which the vocalization start instruction was acquired is within the setting period ts(1) to te(1), the control unit 11 determines that the time at which the vocalization start instruction was acquired is included in the vocalization setting period corresponding to the character M(1), and specifies and sets the character "ko" corresponding to the character M(1) as the syllable to be vocalized.

[0038] Next, the control unit 11 outputs an instruction to the DSP to generate an audio signal based on the set pitch "G4" and the pronunciation of the character "ko". In Fig. 4, the time when the instruction to generate an audio signal based on the set pitch "G4" and the pronunciation of the character "ko" is output to the DSP is represented as time ton(1). The DSP of the control unit 11 starts generating an audio signal based on the instruction.

[0039] Next, assume that in a standby state in the sound generation process, a performance signal including a vocalization stop instruction (note end) associated with the pitch "G4" is received. In this case, the control unit 11 sets the pitch "G4" and determines that the performance signal is a vocalization stop instruction. The DSP of the control unit 11 outputs an instruction to stop generating an audio signal based on the vocalization (character "ko") with the set pitch "G4". In FIG. 4, the time when the instruction to stop generating an audio signal based on the set pitch "G4" and the vocalization of the character "ko" is output is represented as time toff(1). The DSP of the control unit 11 stops generating an audio signal based on the instruction. In FIG. 4, the vocalization period ton(1) to toff(1) is the period during which an audio signal based on the vocalization of the pitch "G4" and the character "ko" is generated.

[0040] Next, a case where the count value tc at which the vocalization start instruction is acquired is a period between the vocalization setting period ts(1)-te(1) and the setting period ts(2)-te(2) and is close to the setting period ts(1)-te(1) will be described with reference to FIG. 5. Assume that a performance signal including a vocalization start instruction associated with the pitch "G4" is received in a standby state in the sound generation process. In this case, the control unit 11 sets the pitch "G4" and determines whether the count value tc at the time the start instruction is acquired is included in the vocalization setting period. Since the time at which the start instruction is acquired is not included in any of the vocalization setting periods corresponding to each character M(i), the control unit 11 calculates the center time tm(i) from the setting periods set immediately before and after the count value tc.

[0041] If the count value tc when the start instruction is acquired is between the set period ts(1) to te(1) and the set period ts(2) to te(2), the control unit 11 calculates the center time tm(1) between the stop time te(1) and the start time ts(2). Here, tm(1)=(te(1)+ts(2)) / 2 is obtained. Next, since the count value tc when the start instruction is acquired is before the center time tm(1), the control unit 11 specifies and sets the character "ko" (character M(1)) of the set period before the center time tm(1) as the syllable to be pronounced. The vocalization period ton(1) to toff(1) is the period during which an audio signal based on the vocalization of the pitch "G4" and the character "ko" is generated.

[0042] Next, a case where the count value tc at which the instruction to start utterance is acquired is a period between the set period ts(1)-te(1) of utterance and the set period ts(2)-te(2) of utterance and is close to the set period ts(2)-te(2) will be described with reference to Fig. 6. If the time at which the instruction to start utterance is acquired is not before the center time tm(1), the control unit 11 specifies and sets the character "ん" (character M(2)) of the set period after the center time tm(1) as the syllable to be pronounced. The period ton(1)-toff(1) is the period during which an audio signal based on the pitch "G4" and the character "ん" is generated.

[0043] In this way, a syllable corresponding to the set period to which the timing determined as the note onset belongs or the set period closest to the timing is identified as the syllable corresponding to the timing determined as the note onset.

[0044] Next, the sound generation process will be described. In the sound generation process, based on a performance operation on the performance operation unit 15, an instruction to generate an audio signal corresponding to each syllable or an instruction to stop generating an audio signal are output.

[0045] 7 is a flowchart showing the sound generation process. This process is realized by the CPU 11a by loading a control program stored in the ROM 11b into the RAM 11c and executing it. This process is started when the user instructs playback of a piece of music.

[0046] The control unit 11 acquires lyrics data from the storage unit 14 (step S101). Next, the control unit 11 executes an initialization process (step S102). In this initialization, a count value tc=0 is set, and various register values ​​and flags are set to their initial values.

[0047] Next, the control unit 11 sets the count value tc=tc+1 and increments the count value tc (step S103). Next, the control unit 11 reads out a portion of the accompaniment data that corresponds to the count value tc (step S104).

[0048] The control unit 11 repeats the processing of steps S103 and S104 until it detects either the end of reading of the accompaniment data, the input of an instruction to stop playing the music piece by the user, or the reception of a performance signal (step S105; No, step S106; No, step S107; No). This repetitive state is the standby state. As described above, the initial value of the count value tc is 0 and corresponds to the timing at which playback of the music piece starts. The control unit 11 increments the count value tc to measure the time based on the timing at which playback of the music piece starts.

[0049] When the reading of the accompaniment data is terminated by reading the accompaniment data to the end in the standby state (step S105; Yes), the control unit 11 terminates the sound generation process. When the user inputs an instruction to stop playing the music in the standby state (step S106; Yes), the control unit 11 terminates the sound generation process.

[0050] When a performance signal is received from the performance operation unit 15 in the standby state (step S107; Yes), the control unit 11 executes instruction processing for generating an audio signal by the DSP (step S108). Details of the instruction processing for generating an audio signal will be described later with reference to FIG. 11. When the instruction processing for generating an audio signal is completed, the control unit 11 returns to step S103 and enters a standby state in which steps S103 and S104 are repeated.

[0051] 8 and 9 show examples of notes generated by playing and syllables assigned to each note. FIG. 8 shows the relationship between a part of the musical score of the main melody of a piece of music designated by the user and notes generated by playing and assigned syllables. Here, the notes assigned with syllables are called syllable notes VN1 to VN7. The notes SN1 to SN9 correspond to syllables in the lyrics data. FIG. 8 shows ideal syllable notes VN1 to VN7 when the user faithfully plays in time with the notes SN1 to SN9. To play faithfully at the right timing means to play so that the notes start at the timing corresponding to the corresponding set period.

[0052] For example, the notes SN1, SN2, and SN3 correspond to the syllables "dra," "gon," and "night." If a user plays the notes SN1, SN2, and SN3 in time with the accompaniment, the syllable notes VN1, VN2, and VN3 are pronounced and generated as notes. The syllable notes VN1, VN2, and VN3 are assigned the syllables "dra," "gon," and "night," respectively. Here, the notes SN3 and SN4 are tied together, so they correspond to one syllable note VN3 "night."

[0053] Similarly, if the user plays the notes SN5 to SN9 in time with the accompaniment, the syllable notes VN4 to VN7 are generated as notes. Here, the notes SN7 and SN8 are tied together, so they correspond to the single syllable note VN6 "night."

[0054] The syllable notes are generated in real time. However, the generated syllable notes can be saved as singing synthesis data including syllable information. Alternatively, the generated syllable notes can be converted to MIDI data that does not include syllable data and saved.

[0055] Fig. 9 shows an example of syllable note generation and a countermeasure example. Case A shows an undesirable example, and case B shows an example in which a countermeasure has been taken against case A. Case B is realized by the instruction process (Fig. 11) described later.

[0056] Case A shows an example where a single long syllable note VN101 was generated even though the user intended to play notes SN2 and SN3 consecutively. Two syllable notes VN2 and VN3 should have been generated, but the volume detected between notes SN2 and SN3 was not sufficiently decreased to separate the syllable notes.

[0057] This type of event is likely to occur when the note value of the preceding note in the series of consecutive homophonic notes is short, such as when a sixteenth note SN2 is followed by a note SN3 of the same pitch. An example of the occurrence of such an event is described with reference to FIG.

[0058] 10 is a diagram showing an example of the acquired change in volume. CC11 is a control change indicating a change (expression) in volume.

[0059] After the volume exceeds the first threshold TH1 and is determined to be the start of a note, if the volume changes in a predetermined manner as shown in Fig. 10 without falling below the second threshold TH2, it is not determined to be the end of a note. For example, after the user plays the note SN2 and starts to produce the syllable note VN101, even if the user lowers the blowing pressure and then plays the note SN3, the volume does not fall below the second threshold TH2. Then, as shown in Example A, "gon" is assigned to the syllable note VN101, and the next syllable "night" is not assigned.

[0060] In this embodiment, in such a case, as shown in case B, the control unit 11 separates the syllable note VN101 into two syllable notes, syllable note VN102 and syllable note VN103, by inserting (placing) a virtual note break in the syllable note VN101. Although not shown in the figure, the end timing of the syllable note VN102 and the start timing of the syllable note VN103 are simultaneous. In other words, when the control unit 11 determines that the volume has changed in the above-mentioned predetermined manner, it determines that the note has ended and that the note has started at the timing when the volume has changed in the predetermined manner, regardless of the result of comparing the volume with the thresholds (thresholds TH1 and TH2). In other words, the control unit 11 determines that the note has ended and started at the same time, thereby substantially inserting a virtual note break.

[0061] In order to distinguish between intentional continuation and intentional division by the user, the predetermined modes shown in Fig. 10 are defined as follows, and the information is stored in the ROM 11b. If the predetermined modes are met, it can be determined that the user is intentionally trying to divide the note.

[0062] The above-mentioned predetermined mode is a mode in which the volume drops at a speed faster than the first predetermined speed within the first predetermined time dur2, and then drops at a speed faster than the second predetermined speed before the second predetermined time dur23 has elapsed from the start of the drop. From the end of the ascent, T4 The increase continues beyond the third predetermined time dur3.

[0063] Here, the first predetermined time dur2 is the time from the descent start time T1 to the descent end time T2, and is, for example, a value in the range of 20 to 100 ms. The second predetermined time dur23 is the time from the descent start time T1 to the ascent end time T4, and is, for example, 200 ms. The third predetermined time dur3 is ,example For example, it is 10 ms. The first and second predetermined speeds are, for example, 0.5 CC / ms. The first and second predetermined speeds do not have to be the same value. These values ​​are not limited to those exemplified. Also, these values ​​may be changed according to the playback tempo. Note that the minimum volume CCx in the case where the above-mentioned predetermined aspect is satisfied is usually higher than the first threshold value TH1.

[0064] FIG. 11 is a flowchart showing the instruction process executed in step S108 of FIG.

[0065] First, in step S201, the control unit 11 compares the volume indicated by the acquired performance signal with the second threshold TH2 to determine whether or not the note has ended (a command to end vocalization). Here, if the volume falls below the second threshold TH2, it is determined that the note has ended. If it is determined that the note has not ended, then in step S202, the control unit 11 determines whether or not the note has started (a command to start vocalization). Here, if the volume exceeds the first threshold TH1, it is determined that the note has started.

[0066] If it is determined in step S201 that the note has ended, in step S210, the control unit 11 instructs the DSP to end the sounding of the syllable being sounded at the timing corresponding to the end of the current note, and ends the process shown in Fig. 11. That is, the control unit 11 outputs an instruction to the DSP to stop the generation of the audio signal started in step S206 or step S209, which will be described later. Therefore, the sound being sounded is muted.

[0067] If it is determined in step S202 that the note is not the start of a note, the control unit 11 executes other processes in step S211 and then proceeds to step S207. In the other processes, for example, when the volume acquired during pronunciation changes without falling below the second threshold value TH2, the control unit 11 outputs an instruction to change the pronunciation volume in response to the change.

[0068] If it is determined in step S202 that the note is to start, the control unit 11 sets the pitch indicated by the acquired performance signal in step S203. In step S204, the control unit 11 executes other processes. In the other processes, the control unit 11 executes a process of continuing pronunciation at the changed pitch without terminating the pronunciation of the syllable being pronounced when the pitch is switched within the same set period, for example. For example, the pronunciation of the syllable "night" is continued at the switched pitch. A long sound "-" is assigned to the syllable note corresponding to the switched pitch.

[0069] In step S205, the control unit 11 identifies a syllable corresponding to the timing determined as the start of the current note by the method described in Fig. 4 to Fig. 6. In step S206, the control unit 11 instructs the DSP to start producing the identified syllable at a pitch and timing corresponding to the start of the current note. That is, the control unit 11 outputs an instruction to the DSP to start generating an audio signal based on the set pitch and the pronunciation of the identified syllable.

[0070] In step S207, the control unit 11 determines whether the volume indicated by the performance signal has changed in the above-mentioned predetermined manner (FIG. 10). If it is determined that the volume has changed in the above-mentioned predetermined manner, the control unit 11 further determines whether the set period to which the timing determined to be the start of the current note belongs is different from the set period to which the timing determined that the volume has changed in the above-mentioned predetermined manner belongs. If the control unit 11 satisfies the conditions that the volume has changed in the above-mentioned predetermined manner and the two set periods are different from each other, it proceeds to step S208, but if these conditions are not satisfied, it ends the process shown in FIG. 11.

[0071] In step S208, the control unit 11 instructs to end pronunciation of the currently specified syllable at a timing corresponding to the end of the current note. In step S209, the control unit 11 instructs to start pronunciation of the next syllable at the same pitch as the pitch of the previous pronunciation (pronunciation started immediately before in S206). Therefore, in steps S208 and S209, regardless of the result of comparing the volume with the threshold, it is determined that the note ends and the note starts at the timing when the volume changes in a predetermined manner, and a virtual note division is inserted (Case B).

[0072] Therefore, on condition that the set period to which the timing determined as the note start belongs is different from the set period to which the timing determined as the volume changed in a predetermined manner belongs, the timing at which the volume changed in a predetermined manner is determined as the note end and the note start. In other words, even if the volume changes in a predetermined manner within the same set period, a virtual note division is not inserted. Therefore, it is possible to prevent division at a point where division is not necessary. After step S209, the control unit 11 ends the process shown in FIG. 11.

[0073] According to this embodiment, the control unit 11 determines the start and end of a note based on the result of comparing the volume of the acquired performance signal with a threshold and the change in pitch of the performance signal (S201, S202). When the control unit 11 determines that the volume has changed in a predetermined manner (FIG. 10), it determines the end of the note and the start of the note at the timing when the volume has changed in the predetermined manner, regardless of the result of comparing the volume with the threshold. Therefore, it is possible to determine the start and end of a note according to the user's intention.

[0074] In particular, by setting the predetermined aspect as shown in FIG. 10, it is possible to distinguish between an intentional continuation and an intentional division by the user, and to appropriately determine the start and end of a note.

[0075] Furthermore, since the note is generated based on the determination of the note start and note end, it is possible to generate the note in accordance with the user's intention.

[0076] Furthermore, when the control unit 11 produces a singing sound obtained by synthesizing syllables, it starts producing the specified syllable at a pitch and timing corresponding to the start of the note, and ends the production at a timing corresponding to the end of the note. This allows for real-time singing production. Moreover, the syllable corresponding to the set period to which the timing determined as the start of the note belongs, or the set period closest to the above-mentioned timing, is specified as the syllable corresponding to the timing determined as the start of the note. This allows for the production of syllables according to the user's intention.

[0077] (Second embodiment) In the second embodiment of the present invention, in the lyrics data (FIG. 2), characters M(i)=M(1) to M(n) are pronounced in order. Therefore, in the sound generation process (FIG. 7), the timing data that specifies the set period for pronunciation in the lyrics data may be ignored. Alternatively, the timing data may be eliminated from the lyrics data.

[0078] In the above-mentioned first embodiment, in the lyrics data, a plurality of syllables to be pronounced are arranged in time series, and each syllable is associated with a set period consisting of a start time and a stop time of pronunciation. Then, the syllable specification method shown in Figs. 4 to 6 is adopted. Therefore, even if the performance timing deviates from the original set period, the syllable corresponding to the set period to which the performance timing belongs (or is close to it) is specified and pronounced. In contrast, in this embodiment, the syllables are specified in order according to the progress of the performance.

[0079] In this embodiment, the instruction process executed in step S108 in Fig. 7 is performed by the process shown in Fig. 12 instead of that shown in Fig. 11. Fig. 12 is a flowchart showing the instruction process executed in step S108 in Fig. 7. In Fig. 12, the same process steps as in Fig. 11 are assigned the same step numbers. Step S204 has been eliminated.

[0080] In this embodiment, in the initialization process in step S102 of FIG. 7, in addition to the process in the first embodiment, the control unit 11 sets the character count value i in M(i) to 1 (character M(i)=M(1)) and sets ts=0. As described above, "i" indicates the vocalization order of syllables in lyrics. ts is the time when the instruction to start the most recent vocalization was obtained. Therefore, the control unit 11 increments "i" to advance the syllables indicated by M(i) one by one among the syllables constituting the lyrics. The other process steps in FIG. 7 are as described in the first embodiment.

[0081] The processes of steps S201 to S203, S210, and S211 in Fig. 12 are as described in the first embodiment. In step S205, the control unit 11 specifies the syllable indicated by the character M(i) as the current syllable. Thus, the syllables are specified in the order in which they are arranged in the lyrics data. In step S206, the control unit 11 instructs to start pronunciation of the specified syllable at a pitch and timing corresponding to the start of the current note.

[0082] In step S207, the control unit 11 determines whether the volume indicated by the performance signal has changed in the above-mentioned predetermined manner (FIG. 10). Here, the present embodiment differs from the first embodiment in that the set period is not taken into consideration. If the volume has changed in the above-mentioned predetermined manner, the control unit 11 proceeds to step S208, and if the volume has not changed in the above-mentioned predetermined manner, the control unit 11 ends the process shown in FIG.

[0083] In step S208, the control unit 11 instructs to end the pronunciation of the first syllable (the syllable whose pronunciation started in the previous step S206) identified this time. In step S209, the control unit 11 instructs to start pronouncing the second syllable following the first syllable at the same pitch as the pitch at which the first syllable was pronounced (the syllable whose pronunciation started in the previous step S206). The second syllable is the syllable following a certain syllable (the first syllable) in the order of the lyrics data. This process determines the user's intention from the change in pitch and inserts a virtual note division. Therefore, it is possible to pronounce the syllable according to the user's intention. After step S209, the control unit 11 ends the process shown in FIG. 12.

[0084] According to this embodiment, the end of a note is determined and the start of a note is determined at the timing when the volume changes in a predetermined manner, and therefore, the same effect as in the first embodiment can be achieved in determining the start and end of a note in accordance with the user's intention.

[0085] Furthermore, since syllables are identified and pronounced in order as the performance progresses, it is possible to pronounce syllables in accordance with the user's intention.

[0086] (Third embodiment) In the third embodiment of the present invention, syllables are not specified or pronounced, but rather, sounds (for example, wind instrument sounds) are pronounced in accordance with the progress of the performance. Therefore, lyric data (FIG. 2) is not essential. Furthermore, music data including accompaniment data is not essential.

[0087] In this embodiment, the instruction process executed in step S108 in Fig. 7 is the process shown in Fig. 13 instead of Fig. 11. Fig. 13 is a flowchart showing the instruction process executed in step S108 in Fig. 7. In Fig. 13, the same process steps as in Fig. 11 are assigned the same step numbers. Steps S204 and S205 have been eliminated.

[0088] The processing of steps S201 to S203 and S211 in FIG. 12 is as described in the first embodiment. In step S206, the control unit 11 issues an instruction to start sounding at a pitch and timing corresponding to the start of the current note. The sounding tone may be a predetermined one. In addition, in other processing of step S211, the sounding tone may be switched according to a user operation. In step S210, the control unit 11 issues an instruction to end the sounding of the sound being sounded at a timing corresponding to the end of the current note, and ends the processing shown in FIG. 13.

[0089] In step S207, the control unit 11 determines whether the volume indicated by the performance signal has changed in the above-mentioned predetermined manner (FIG. 10). Here, the present embodiment differs from the first embodiment in that the set period is not taken into consideration. If the volume has changed in the above-mentioned predetermined manner, the control unit 11 proceeds to step S208, and if the volume has not changed in the above-mentioned predetermined manner, the control unit 11 ends the process shown in FIG.

[0090] In step S208, the control unit 11 instructs the end of the sound being produced. In step S209, the control unit 11 instructs the start of production again at the same pitch as the pitch at which the sound was produced in the previous step S206. This process determines the user's intention from the change in pitch and inserts a virtual note division. After step S209, the control unit 11 ends the process shown in FIG.

[0091] According to this embodiment, the end of a note is determined and the start of a note is determined at the timing when the volume changes in a predetermined manner. Therefore, the same effect as in the first embodiment can be achieved in determining the start and end of a note according to the user's intention. Also, real-time pronunciation that is not a syllabic pronunciation and reflects the user's intention can be realized.

[0092] In the above embodiments, the sound control device 100 may be any other wind instrument with a breath sensor as long as it can acquire the volume as a continuous quantity. Alternatively, the sound control device 100 is not limited to a wind instrument type, and may be in other forms such as a keyboard instrument. For example, when the present invention is applied to a keyboard instrument, each key may be provided with an after-sensor or other function, and the volume may be configured to change continuously in response to key operation. Alternatively, an electronic instrument connected to a volume pedal may be used, and volume information may be input by operating the volume pedal.

[0093] The performance signal (performance information) may be acquired from an external device via communication. Therefore, it is not essential to provide the performance operation unit 15, and it is not essential for the sound control device 100 to have the functions and form of a musical instrument.

[0094] It is not essential to provide the pronunciation unit 18. The synthesized singing sound and information on the sound to be pronounced may be transmitted to the outside via communication and pronounced by an external device having the same function as the pronunciation unit 18. At least one function of the generation unit 33, the identification unit 34, the singing sound synthesis unit 35, and the instruction unit 36 ​​may be provided in an external device connected via the communication I / F 19.

[0095] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0096] In addition, a storage medium storing a control program represented by software for achieving the present invention may be read into the device to achieve the same effect as the present invention. In that case, the program code itself read from the storage medium realizes the novel function of the present invention, and the non-transient computer-readable storage medium storing the program code constitutes the present invention. The program code may also be supplied via a transmission medium, etc., in which case the program code itself constitutes the present invention. In addition to ROM, the storage medium in these cases may be a floppy disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a non-volatile memory card, etc. In addition, the non-transient computer-readable storage medium may also include a volatile memory (e.g., DRAM (Dynamic Random Access Memory)) inside a computer system that is a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, which holds the program for a certain period of time. [Explanation of symbols]

[0097] 11 Control section 31 Acquisition Department 32 Judgment section 33 Generation part 34 Specific part 35 Singing sound synthesis department 36 Instruction section 100 Sound control device

Claims

1. an acquisition unit that acquires performance information including pitch information indicating a pitch and volume information indicating a volume detected as a continuous quantity; a determination unit that determines a note start and an note end based on a comparison result between a volume in the performance information and a threshold value and a change in pitch in the performance information, when it is determined that the volume has changed in a predetermined manner, the determination unit determines that the note has ended and that the note has started at the timing when the volume has changed in the predetermined manner, regardless of a result of comparing the volume with the threshold value; The specified aspect of the sound control device is that the volume decreases at a speed faster than a first specified speed within a first specified time, and then, before a second specified time has elapsed from the start of the descent, continues to increase at a speed faster than the second specified speed and for more than a third specified time from the start of the rise.

2. The sound control device according to claim 1 , further comprising a generating unit configured to generate a note based on the determination of the note start and the note end by the determining unit.

3. The sound control device according to claim 1 , wherein the determination unit determines that a note has started when the volume exceeds a first threshold value.

4. The sound control device according to claim 3 , wherein the determination unit determines that the note has ended when the volume falls below a second threshold after determining that the note has started.

5. The sound control device according to claim 3 , wherein the determination unit is configured to determine that the note has ended and that the note has started when the pitch has changed while the volume has exceeded the first threshold value.

6. an identification unit that identifies a syllable corresponding to the timing determined by the determination unit to be the start of a note from lyric data in which a plurality of syllables to be pronounced are arranged in time series; 6. The sound control device according to claim 1, further comprising an instruction unit that instructs the syllable identified by the identification unit to start being pronounced at a pitch and timing corresponding to the start of the note, and to end being pronounced at a timing corresponding to the end of the note.

7. 7. The sound control device according to claim 6, wherein, after instructing the start of pronunciation of a first syllable corresponding to the start of the note, if the volume has changed in the predetermined manner to determine the end of the note and the start of the note, the instruction unit instructs the end of pronunciation of the first syllable and instructs the start of pronunciation of a second syllable following the first syllable at the same pitch as the pitch at which the first syllable was pronounced.

8. In the lyrics data, each of the syllables is associated with a set section consisting of a pronunciation start time and a pronunciation stop time, The sound control device according to claim 6 , wherein the identification unit identifies a syllable corresponding to a set section to which the timing determined to be a note start belongs or a set section closest to the timing as the syllable corresponding to the timing determined to be a note start.

9. The sound control device according to claim 8, wherein the determination unit determines the note end and the note start at the timing when the volume changes in the specified manner, under the condition that the set section to which the timing determined to be the note start belongs is different from the set section to which the timing determined to be the volume change in the specified manner belongs.

10. The sound control device according to claim 6 , further comprising a sound generating unit that generates a singing sound by synthesizing syllables based on an instruction by the instruction unit.

11. 6. The sound control device according to claim 1, further comprising an instruction unit that instructs sound production to start at a pitch and timing corresponding to the note start, and that instructs sound production to end at a timing corresponding to the note end.

12. A sound control device according to any one of claims 1 to 11, a performance operation unit for a user to input the performance information.

13. The performance operation unit includes a breath sensor that detects pressure changes, The electronic musical instrument according to claim 12 , wherein the volume information is acquired based on a pressure change detected by the breath sensor.

14. A program for causing a computer to execute a control method for a sound control device, The control method of the sound control device includes: Acquire performance information including pitch information indicating a pitch and volume information indicating a volume detected as a continuous quantity; determining a note start and an end based on a comparison result between the volume of the performance information and a threshold value and a change in pitch of the performance information; when it is determined that the volume has changed in a predetermined manner, regardless of a result of comparing the volume with the threshold value, the timing at which the volume has changed in the predetermined manner is determined as the end of the note and the start of the note; The specified aspect is that the volume decreases at a speed faster than a first specified speed within a first specified time, and then, before a second specified time has elapsed from the start of the descent, continues to increase at a speed faster than the second specified speed and for a period exceeding a third specified time from the start of the increase, said program.

15. A control method for a sound control device implemented by a computer, comprising: Acquire performance information including pitch information indicating a pitch and volume information indicating a volume detected as a continuous quantity; determining a note start and an end based on a comparison result between the volume of the performance information and a threshold value and a change in pitch of the performance information; when it is determined that the volume has changed in a predetermined manner, regardless of a result of comparing the volume with the threshold value, the timing at which the volume has changed in the predetermined manner is determined as the end of the note and the start of the note; A control method for a sound control device, wherein the specified aspect is that the volume decreases at a speed faster than a first specified speed within a first specified time, and then, before a second specified time has elapsed from the start of the descent, continues to increase at a speed faster than the second specified speed and for more than a third specified time from the start of the increase.

Images