Performance control device, performance control method, and performance system

Abstract

To improve the accuracy of performance based on user vocalizations. [Solution] The performance control device according to the present invention is characterized by having an acquisition unit that acquires the sound emitted by a performer who performs using multiple sound-producing instruments in order to play some of the sound-producing instruments via a microphone, and acquires the movement of the performer's throat via a predetermined detection device, and an output unit that outputs the sound emitted by some of the multiple sound-producing instruments played by the performer based on the sound acquired by the acquisition unit and the movement of the throat.

Description

Technical Field

[0004] ,

[0006] , , , ,

[0005] , , , , ,

[0001] The present invention relates to a performance control device, a performance control method, and a performance system.

Background Art

[0002] Conventionally, a technique for outputting a sound corresponding to a sound uttered by a user has been known. As an example of such a technique, there is known a technique in which a sound effect word uttered by a user is picked up by a microphone, and a sound of a musical instrument corresponding to the picked-up sound effect word among a plurality of previously recorded musical instrument sounds is output (for example, Patent Document 1). Also, a technique for outputting a sound effect corresponding to a sound effect word uttered by a user among a plurality of sound effects has been known (for example, Patent Document 2).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the above-described technique, when a sound effect word is misrecognized, a performance contrary to the user's intention may be performed. For example, in the above-described technique, when a sound having characteristics similar to those of a sound effect word is picked up, the picked-up sound may be misrecognized as a sound effect word, and there is a risk that a sound of a musical instrument is output even though the user has not uttered a sound effect word.

[0005] The present application has been made in view of the above, and an object thereof is to improve the accuracy of performance based on the user's utterance.

Means for Solving the Problems

[0006] The performance control device according to the present invention is characterized by having an acquisition unit that acquires, via a microphone, the sound emitted by a performer who performs using multiple sound-producing devices in order to play some of the sound-producing devices, and also acquires the movement of the performer's throat via a predetermined detection device, and an output unit that outputs the sound emitted by some of the multiple sound-producing devices played by the performer based on the sound acquired by the acquisition unit and the movement of the throat. [Effects of the Invention]

[0007] According to one embodiment, the accuracy of performance based on the user's vocalizations can be improved. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 shows an example of the performance control process according to the embodiment. [Figure 2] Figure 2 shows an example of the configuration of a performance system according to the embodiment. [Figure 3] Figure 3 shows an example of an output sound database. [Figure 4] Figure 4 shows an example of a condition database. [Figure 5] Figure 5 is a flowchart showing an example of the procedure for the performance control process according to this embodiment. [Modes for carrying out the invention]

[0009] The following describes in detail, with reference to the drawings, embodiments for implementing the performance control device, performance control method, and performance system according to the present application (hereinafter referred to as "embodiments"). Note that these embodiments do not limit the performance control device, performance control method, and performance system according to the present application. Furthermore, the same parts are denoted by the same reference numerals in each of the following embodiments, and redundant descriptions are omitted.

[0010] [1. Embodiments] The information processing realized by the performance control device 100 of this embodiment will be explained using Figure 1. Figure 1 is a diagram showing an example of the performance control processing according to the embodiment. In the following explanation, the user who plays the musical instrument will be referred to as the performer.

[0011] First, let's explain the example in Figure 1. Figure 1 shows performer P1 playing a drum set DS. A drum set DS consists of multiple instruments (so-called percussion instruments) such as a snare drum, bass drum, cymbals, and hi-hat, each of which can be operated independently, and is played simultaneously by the same performer. In the following explanation, each of the multiple instruments played by the same performer will be referred to as a sound-producing instrument. In the example in Figure 1, a drum set DS is shown as an example composed of sound-producing instruments such as a hi-hat DS1, a snare drum DS2, and a bass drum DS3. In addition to these sound-producing instruments, a drum set DS may also include various other sound-producing instruments such as cymbals in any number and combination.

[0012] Here, the bass drum DS3 has a speaker SP installed inside it. More specifically, the speaker SP is installed inside the bass drum DS3 so that the bass drum DS3 vibrates when the sound it emits is produced. As will be described later, the speaker SP outputs pre-recorded sounds from a sound generator (for example, the sound of actually playing the hi-hat DS1, snare drum DS2, or bass drum DS3, or sounds generated to have the characteristics of the instrument's sound, synthesized sounds, etc.). As a result, when the sound of the bass drum DS3 is emitted from the speaker SP, the performer P1 can feel the resonance of the bass drum DS3 and get the sensation that they are actually playing the bass drum DS3.

[0013] The microphone MC is a sound-collecting microphone positioned near the performer P1's mouth to detect the voice that performer P1 makes for the performance. While Figure 1 shows an example where the microphone MC is positioned away from performer P1, the microphone MC could also be a microphone worn by performer P1.

[0014] Sensor SE is a sensor that detects information related to the performance behavior of performer P1, and is a sensor that detects a different type of information than microphone MC. For example, Sensor SE is a biosensor that is placed in the throat of performer P1 and detects the bio-vibrations of performer P1. For example, Sensor SE can be implemented using an acceleration sensor or a sensor that detects electromyography.

[0015] When playing the drum set DS, performer P1 can use their right hand, left hand, right foot, and left foot to play various sound-producing instruments, allowing them to produce up to four sounds simultaneously. However, if performer P1 has any impairment in their limbs, the number of sounds they can produce simultaneously will decrease, making it impossible to play as intended. For example, if performer P1 cannot move their right foot properly, they will not be able to properly play the bass drum DS3, which is played with the right foot, among the sound-producing instruments that make up the drum set DS. Furthermore, some performers P1 may want to produce five or more sounds simultaneously.

[0016] Therefore, the performance control device 100 assists the performer P1 in playing the drum set DS based on the sound acquisition results from the microphone MC and the detection results from the sensor SE. The output processing of drum sounds performed by the performance control device 100 will be explained below using Figure 1. In the example in Figure 1, the performance control device 100 acquires the audio information acquired by the microphone (step S1). For example, in the example shown in Figure 1, performer P1 makes the sound V "do-do-chi" as a performance action to play the bass drum DS3. In such a case, the performance control device 100 sequentially acquires the waveform of the sound detected by the microphone as audio information when performer P1 makes the sound V "do-do-chi".

[0017] Further, the performance control device 100 acquires the motion information acquired by the biological sensor (step S2). For example, the performance control device 100 acquires, as the motion information, information indicating the movement M of the throat accompanying the voice of the performer P1. To give a more specific example, the performance control device 100 acquires information indicating the movement of the Adam's apple as the movement M of the throat accompanying the voice V. Note that the performance control device 100 may acquire the voice information and the motion information in any order, or may acquire them simultaneously in parallel.

[0018] Subsequently, the performance control device 100 determines whether the voice information satisfies a predetermined first condition and whether the motion information satisfies a predetermined second condition (step S3). Hereinafter, the specific content of the determination process executed by the performance control device 100 as step S3 will be described.

[0019] First, the performance control device 100 determines whether the voice information satisfies a predetermined first condition. More specifically, for each voice information, the performance control device 100 determines whether the sound uttered by the performer P1 includes predetermined sound characteristics (for example, volume, frequency characteristics, etc.). For example, when the predetermined voice corresponding to the bass drum DS3 is "do", the performance control device 100 determines whether the voice information includes the characteristics of the sound corresponding to the voice "do". Then, when the voice information includes the characteristics of the sound corresponding to the voice "do", the performance control device 100 determines that the voice information satisfies the predetermined first condition.

[0020] Here, it is conceivable that the performer P1 has a desire to utter a voice not only for uttering a voice such as "do" to play the bass drum DS3 but also for uttering a voice for taking a rhythm such as "chick". Therefore, when only the voice "chick" is included in the voice information or when the voice information does not include the characteristics of the sound corresponding to the voice "do", the performance control device 100 determines that the voice information does not satisfy the predetermined first condition.

[0021] By repeatedly performing this process, the performance control device 100 can appropriately acquire the voice that performer P1 makes for playing the bass drum DS3 from the audio information picked up by the microphone MC. In addition, the sound picked up by the microphone MC will include not only the voice of performer P1, but also the sound of the snare drum DS2 played by performer P1 with his right or left hand, the sound of the hi-hat DS1 played with his left foot, and even the sounds of instruments played by other performers. Even in such cases, the performance control device 100 can appropriately determine performer P1's performance actions by detecting the voice that performer P1 makes for playing from the sound picked up by the microphone MC. For example, if performer P1 makes a sound like "do-do-chi", the performance control device 100 will determine that the first condition is met at the timing of the first two "do" sounds, and that the first condition is not met at the timing of the "chi" sound.

[0022] Furthermore, the performance control device 100 determines whether the operation information satisfies a predetermined second condition. To give a more specific example, the performance control device 100 determines whether the operation information includes a movement M of the throat that was previously measured when performer P1 uttered a sound such as "do". In other words, the performance control device 100 determines, as the first condition, whether the sound that performer P1 utters for performance is included, and as the second condition, whether the movement of performer P1's throat is the movement of the throat when uttering a sound for performance. To put it another way, the performance control device 100 determines whether the performance action related to the performance action captured by the microphone MC has been detected by a sensor SE that detects a different type of information than that of the microphone MC. Then, the performance control device 100 determines that the operation information satisfies the predetermined second condition if the movement M of the throat that performer P1 uttered a sound such as "do" is included in the operation information. On the other hand, the performance control device 100 determines that the performance information does not satisfy a predetermined second condition if the movement M of the throat when performer P1 makes a sound such as "do" is not included in the performance information (for example, if only the movement of the throat when making a sound such as "tsk" is included in the performance information). For example, the sound "do" involves a large vibration of the throat, while the sound "tsk" involves a small vibration of the throat. Therefore, the performance control device 100 can determine whether or not performer P1 is making a sound such as "do" by determining whether or not the signal level of the vibration detected by sensor SE exceeds a predetermined threshold.

[0023] Then, if each piece of information satisfies the conditions, the performance control device 100 outputs the sound of a predetermined sound-producing device from among the multiple sound-producing devices to be played (step S4). For example, if the voice information satisfies a predetermined first condition and the operation information satisfies a predetermined second condition, the performance control device 100 outputs a pre-recorded bass drum DS3 sound "don" from the speaker SP via the amplifier AP. For example, the performance control device 100 outputs the bass drum DS3 sound "don" at the moment when performer P1 makes a sound such as "do". Therefore, if performer P1 makes a sound V "do-do-chi", the performance control device 100 outputs the bass drum DS3 sound "don-don" from the speaker SP. If the sound detected by the microphone MC includes sounds other than the voice of performer P1, it may be difficult to appropriately detect the voice of performer P1 from the voice information detected by the microphone MC. Therefore, the performance control device 100 can appropriately detect the performance actions of performer P1 by using the information detected by the sensor SE in addition to the detection result by the microphone MC.

[0024] As described above, the performance control device 100 outputs sound from the sound generator, taking into account not only the voice of the performer P1 but also the movement of the throat associated with the voice production. In other words, the performance control device 100 outputs sound from the sound generator when the voice information picked up by the microphone MC satisfies the first condition and the motion information detected by the sensor SE, which acquires information of a different type than that of the microphone MC, satisfies the second condition. As a result of this processing, the performance control device 100 can prevent malfunctions such as outputting sound from the sound generator at a timing unintended by the performer P1.

[0025] For example, instruments composed of multiple individually playable sound-producing devices, such as a drum set DS, are typically played with all four limbs. However, if a performer P1 has difficulty moving some of their limbs, it becomes difficult to play such instruments as intended. Furthermore, if performer P1 previously played using all four limbs, but due to illness or an accident, some of their limbs (for example, only the right foot, only the left hand, etc.) become immobile, it becomes difficult to play with the remaining limbs because they still have an image of how they previously played.

[0026] Therefore, the performance control device 100 outputs the sound of a predetermined sound-producing device (more specifically, a sound-producing device played by a limb that cannot be moved properly) from among the multiple sound-producing devices that make up the instrument, based on the sound acquisition results by the microphone MC and the detection results by the sensor SE. More specifically, the performance control device 100 outputs the sound of the sound-producing device based on the user's voice and the movement of the throat associated with the voice production. As a result of this processing, the performance control device 100 can appropriately assist the performer P1 in playing without significantly changing the playing method using the remaining limbs, even if the performer P1 cannot move some of their limbs properly. Furthermore, since the performance control device 100 controls the sound of the sound-producing device by considering not only the sound acquisition results by the microphone MC but also the detection results by the sensor SE, it can prevent malfunctions due to false detection of the performer P1's voice.

[0027] Furthermore, the performance assistance provided by the performance control device 100 is useful not only for performers P1 who have difficulty moving some of their limbs, but also for performers P1 who have no problems with their limbs but wish to broaden their performance range. For example, if performer P1 wants to play five or more of the sound-producing devices that make up the drum set DS simultaneously, the performance control device 100 can enable the performance of sound-producing devices other than the four that are played with the limbs by outputting the sound of a predetermined sound-producing device based on the voice and throat movements of performer P1 who is playing with their limbs. In this way, the performance control device 100 can provide performer P1 with the function of a fifth set of limbs.

[0028] Furthermore, in the above embodiment, the speaker SP is installed inside the bass drum DS3, and the sound of the bass drum DS3 is output from the speaker SP. This allows the bass drum DS3 to vibrate with the sound output from the speaker SP, providing the performer P1 with a sense of playing and feedback. It also provides the performer P1 with a sense of enjoyment while playing. In this way, the performance control device 100 not only improves the freedom of performance for the performer P1, but also provides the performer P1 with a sense of enjoyment while playing.

[0029] [2. Configuration of the performance system] Next, the configuration of the performance system 1 according to the embodiment will be described using Figure 2. Figure 2 is a diagram showing an example of the configuration of the performance system according to the embodiment. As shown in Figure 2, the performance system 1 includes a microphone MC, a sensor SE, a speaker SP, an amplifier AP, and a performance control device 100.

[0030] The microphone MC, sensor SE, and performance control device 100 are connected via wired or wireless communication. Additionally, the speaker SP, amplifier AP, bass drum DS3, performance device PD, and performance control device 100 are connected via wired or wireless communication.

[0031] The performance system 1 shown in Figure 2 may include at least one of the following: multiple microphones (MC), multiple sensors (SE), multiple speakers (SP), multiple amplifiers (AP), multiple bass drums (DS3), and multiple performance control devices (100). Furthermore, as will become clear in the following description, the performance system 1 may also play sound-producing devices other than the bass drums (DS3).

[0032] A microphone (MC) is a sound recording device that has communication capabilities to communicate with various devices and captures the voice of a performer. Specifically, a microphone (MC) is a microphone that captures the voice emitted by a performer in order to play some of the multiple sound-producing instruments that the performer uses.

[0033] A sensor SE is a detection device that has communication capabilities to communicate with various devices and detects performance behavior related to the sound capture results from a microphone MC. For example, a sensor SE is a biosensor. More specifically, a sensor SE is an accelerometer or an electromyograph sensor. For example, a sensor SE detects the acceleration or velocity of its physical movement based on the acceleration detected by the function of an accelerometer.

[0034] The performance control device 100 is an information processing device capable of communicating with various devices. For example, the performance control device 100 may be a desktop PC (Personal Computer), a notebook PC, a tablet terminal, a mobile phone, or a PDA (Personal Digital Assistant). The performance control device 100 may also be implemented by, for example, a server device or a cloud system.

[0035] The amplifier AP is an amplification device that amplifies the audio signal output from the performance control device 100. For example, the amplifier AP amplifies the sound of the bass drum DS3 output by the performance control device 100 and outputs it from the speaker SP.

[0036] [3. Functional Configuration of Information Processing Equipment] Next, the functional configuration of the performance control device 100 will be described. As shown in Figure 2, the performance control device 100 has a communication unit 110, a storage unit 120, and a control unit 130.

[0037] (Regarding Communications Unit 110) The communication unit 110 controls communication between the performance control device 100 and the microphone MC, sensor SE, amplifier AP, etc.

[0038] (Regarding memory unit 120) The storage unit 120 is implemented by, for example, semiconductor memory elements such as RAM (Random Access Memory) and flash memory, or by storage devices such as hard disks and optical discs. As shown in Figure 2, the storage unit 120 has an output sound database 121 and a condition database 122.

[0039] (Regarding output sound database 121) The output sound database 121 stores various information about the sounds emitted from the sound generator, which have been pre-recorded. Here, an example of the information stored in the output sound database 121 will be explained using Figure 3. Figure 3 is a diagram showing an example of the output sound database. In the example in Figure 3, the output sound database 121 has items such as "Sound ID (Identifier)" and "Sound Data".

[0040] The "Sound ID" indicates identification information for identifying sound data. The "Sound Data" is sound data associated with the "Sound ID," and is, for example, data in an uncompressed audio format such as PCM (Pulse Code Modulation). The sound data may also be data in a lossless compressed audio format such as FLAC (Free Lossless Audio Codec). For example, Figure 3 shows an example where the sound data identified by the sound ID "SID-1" is "Data-1." Each "Sound Data" is a recording of a sound emitted from a sound generator. Furthermore, each "Sound Data" may be a recording of the sound produced by the same sound generator using different playing methods, or it may be a recording of sounds emitted from different sound generators.

[0041] (Regarding Condition Database 122) Returning to Figure 2, let's continue the explanation. The condition database 122 stores the first and second conditions. Here, we will use Figure 4 to explain an example of the information stored in the condition database 122. Figure 4 is a diagram showing an example of a condition database. In the example in Figure 4, the condition database 122 has items such as "Condition ID," "First Condition," "Second Condition," and "Control Content." Note that any predetermined conditions can be registered in the condition database 122 as the first and second conditions. Also, the example shown in Figure 4 describes the case where the microphone MC is a microphone and the sensor SE is an acceleration sensor.

[0042] "Condition ID" indicates identification information for identifying the judgment conditions related to information picked up by the microphone MC and information detected by the sensor SE. "First Condition" indicates the judgment condition for audio information picked up by the microphone MC. "Second Condition" indicates the judgment condition for information detected by the sensor SE. "Control Content" indicates the control content when the judgment condition associated with "Condition ID" is met.

[0043] Figure 4 shows an example of a judgment condition identified by condition ID "condition-1," where the first condition is "speech parameter VP1," the second condition is "operation parameter MP1," and the control content is "output SID-1." This judgment condition indicates that if the audio information picked up by microphone MC satisfies "speech parameter VP1" and the operation information detected by sensor SE satisfies "operation parameter MP1," the sound data indicated by sound ID "SID-1" will be output. In the example shown in Figure 4, conceptual values ​​such as "speech parameter VP1" and "operation parameter MP1" are shown. "Speech parameter VP1" is a parameter that indicates the characteristics of the voice that performer P1 makes for performance (e.g., sound pressure or frequency characteristics in each frequency band), and "operation parameter MP1" is a parameter that indicates the characteristics of the movement of performer P1's throat when making a voice corresponding to the associated "speech parameter VP1" (e.g., frequency, amplitude, or electromyography values ​​of the throat surface).

[0044] (Regarding the control unit 130) Returning to Figure 2, let's continue the explanation. The control unit 130 is implemented by, for example, a CPU (Central Processing Unit) or an MPU (Micro Processing Unit) executing various programs stored in the internal memory of the performance control device 100 using RAM as the working area. Alternatively, the control unit 130 may be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). As shown in Figure 2, the control unit 130 according to this embodiment has an acquisition unit 131, a determination unit 132, an output unit 133, and a setting unit 134, and implements or executes the information processing functions and operations described below.

[0045] (Regarding acquisition unit 131) The acquisition unit 131 acquires audio information picked up by the microphone MC and motion information detected by the sensor SE. Specifically, the acquisition unit 131 acquires the sound pickup results from the microphone MC, which detect information about the performance actions of a performer who is performing using multiple sound-producing devices. The acquisition unit 131 also acquires detection results from the sensor SE, which detects information about the performer's performance actions that is of a different type than the information picked up by the microphone MC.

[0046] More specifically, the acquisition unit 131 acquires detection results from a sensor SE that detects information of a different type from that detected by the microphone MC, which is related to performance actions concerning the sound to be detected by the microphone MC. For example, the acquisition unit 131 acquires audio information via the microphone MC that includes the voice V "do-do-chi" uttered by the performer for the purpose of playing. The acquisition unit 131 also acquires motion information, including the movement of the throat, when the performer utters the voice V for the purpose of playing.

[0047] (Regarding the determination unit 132) The determination unit 132 determines whether the detection results of the microphone MC and sensor SE acquired by the acquisition unit 131 satisfy predetermined conditions. More specifically, for each determination condition stored in the condition database 122, the determination unit 132 determines whether the voice information satisfies a predetermined first condition and the operation information satisfies a predetermined second condition. For example, if the determination conditions shown in Figure 4 are registered in the condition database 122, the determination unit 132 determines whether the voice information picked up by the microphone MC satisfies "voice parameter VP1" and whether the operation information detected by the sensor SE satisfies "operation parameter MP1".

[0048] Furthermore, the determination unit 132 determines whether the first and second conditions are met for each determination condition registered in the condition database 122. If the determination unit 132 determines that the first and second conditions are met, it notifies the output unit 133 of the condition ID of the determination condition that it determined to be met. In this way, the determination unit 132 determines whether different conditions are met for the information picked up by the microphone MC and the information detected by the sensor SE, and notifies the output unit 133 if it determines that each piece of information meets the conditions. For example, if the determination unit 132 determines that the first and second conditions are met for multiple determination conditions, it may notify the output unit 133 of one or more determination conditions selected by an arbitrary rule (for example, determination conditions with lower condition IDs or determination conditions with a predetermined higher priority), or it may notify the output unit 133 of all determination conditions that meet the conditions.

[0049] (Regarding output unit 133) Based on the sound acquisition results and detection results acquired by the acquisition unit 131, the output unit 133 outputs sounds emitted by some of the multiple sound-producing devices played by the performer. For example, if the audio information acquired from the microphone MC includes the voice spoken by the performer P1 for the performance, and the motion information acquired from the biosensor SE indicates the movement of the throat when speaking for the performance, the output unit 133 outputs these voices and movements, and the sounds of the sound-producing devices corresponding to them.

[0050] For example, if the determination unit 132 determines that the condition is met and the condition ID is "condition-1", the output unit 133 refers to the condition database 122 and identifies the control content "output SID-1" associated with the condition ID "condition-1". In this case, the output unit 133 refers to the output sound database 121 and identifies the sound data "data-1" associated with the sound ID "SID-1". Then, the output unit 133 outputs the identified sound data "data-1" to the amplifier AP and outputs it from the speaker SP.

[0051] Furthermore, if the output unit 133 receives notification of multiple determination conditions from the determination unit 132, it may output all of the sounds from the sound generators corresponding to these determination conditions. For example, the output unit 133 may refer to the condition database 122 to identify the control content corresponding to the notified condition ID, and then output the sounds from multiple sound generators by executing each of the identified control content.

[0052] (Regarding setting section 134) The setting unit 134 accepts settings for the output sound database 121 or the condition database 122. For example, when the setting unit 134 receives sound data from a sound generator to be output in response to the performance actions of performer P1 via various input devices (not shown) or external terminal devices, it assigns a sound ID to the received sound data and registers it in the output sound database 121. In addition, when the setting unit 134 receives a first condition regarding the sound picked up by the microphone MC, a second condition regarding the information detected by the sensor SE, and the control content when the first and second conditions are met, it associates these and registers them in the condition database 122.

[0053] [4. Information Processing Flow] The information processing procedure of the performance control device 100 according to this embodiment will be explained using Figure 5. Figure 5 is a flowchart showing an example of the performance control processing procedure according to this embodiment.

[0054] As shown in Figure 5, the performance control device 100 acquires the sound pickup result from the microphone MC (step S101). Next, the performance control device 100 acquires the detection result from the sensor SE (step S102). Steps S101 and S102 may be executed in any order. Then, the performance control device 100 determines whether the sound pickup result from the microphone MC satisfies a predetermined first condition (step S103). For example, if the performance control device 100 determines that the sound pickup result from the microphone MC does not satisfy the predetermined first condition (step S103; No), it executes step S101 again.

[0055] On the other hand, if the performance control device 100 determines that the sound pickup result from the microphone MC satisfies a predetermined first condition (step S103; Yes), it determines whether the detection result from the sensor SE satisfies a predetermined second condition (step S104). If the performance control device 100 determines that the detection result from the sensor SE does not satisfy the predetermined second condition (step S104; No), it repeats step S101. On the other hand, if the performance control device 100 determines that the detection result from the sensor SE satisfies the predetermined second condition (step S104; Yes), it outputs sound from the speaker (step S105) and terminates the process. The performance control device 100 may also execute steps S103 and S104 in reverse. Furthermore, the performance control device 100 may repeatedly execute the information processing shown in Figure 5.

[0056] [5. Variations] The above-described embodiment is merely an example, and various modifications and applications are possible. The following describes some variations.

[0057] [5-1. About Sensors] In the above embodiment, an example was described in which the performance behavior of performer P1 is detected by a biosensor, sensor SE, but the embodiment is not limited to this. For example, sensor SE may be a camera that photographs the user. To give a more specific example, the performance control device 100 may identify pre-set performance behaviors of performer P1, such as facial movements, eyelid movements, and eye movements, from the video footage of performer P1, and output a sound corresponding to the identified performance behavior.

[0058] Furthermore, as another example, the sensor SE may be a non-contact sensor. To give a more specific example, the sensor SE may be a sensor that photographs the mouth of performer P1 and detects the movement of the mouth. Alternatively, the sensor SE may be a sensor that optically detects the movement of the throat of performer P1.

[0059] Furthermore, the performance control device 100 may also consider the detection results of other sensors in addition to the microphone MC and sensor SE. In this case, an AND condition of the sound pickup results and detection results from the microphone MC, sensor SE, and additional sensors may be set as the determination condition for the output processing performed by the performance control device 100.

[0060] Furthermore, the sensor SE can be any sensor as long as it provides information correlated with the sound detected by the microphone MC. For example, the sensor SE may be a sensor that detects the electromyographic activity of various muscles, such as the chest or lungs, in addition to the throat of the performer P1. The microphone MC is a microphone placed over the mouth of the performer P1 to detect the sound coming from the performer P1's mouth, and the sensor SE may be a microphone placed in the throat of the performer P1 to detect the sound from the throat. In other words, the sensor SE may be the same type of sensor as the microphone MC, as long as it detects the performance behavior detected by the microphone MC using a different type of information than the microphone MC, and any type of sensor can be used.

[0061] [5-2. About musical instruments] Furthermore, although the above embodiment describes an example in which performer P1 plays a drum set DS consisting of multiple sound-producing instruments (specifically, percussion instruments), the embodiment is not limited to this. For example, the instrument to be played may be a piano or an electronic organ. A piano is an instrument that has multiple keys, and when each key is pressed, a hammer strikes the string corresponding to the key, thereby producing sound. If we consider such keys, hammers, and strings as a single sound-producing instrument, then a piano can be said to be an instrument that includes multiple sound-producing instruments, each capable of independently producing sound. The performance control device 100 may assist in playing not only instruments consisting of multiple sound-producing instruments, each having an independent housing (in other words, a set of instruments), such as a drum set, but also instruments that have a single structure as an instrument, such as a piano or an electronic organ, but include multiple sound-producing instruments, each capable of independently producing sound. The performance control device 100 may also control a single instrument that outputs a single sound according to the information picked up by the microphone MC and the information detected by the sensor SE.

[0062] [5-3. Regarding the controlled object] Furthermore, the performance control device 100 may assist the performer P1's performance by controlling devices other than the sound-producing device based on the information picked up by the microphone MC and the information detected by the sensor SE. To give a more specific example, the performance control device 100 may control the damper pedal, sostenuto pedal, shift pedal, etc., of a piano, or various amplifiers for a guitar. The performance control device 100 may also control whether the hi-hat is open or closed. In addition, the performance control device 100 may control devices other than musical instruments, such as lights.

[0063] [5-4. Regarding sound output] Furthermore, the performance control device 100 may control the output of sound to assist the performance actions of the performer P1. For example, in the embodiment described above, the performance control device 100 output the sound of the bass drum DS3 when the performer P1 made the sound "do" and the corresponding throat movement was detected. However, the performance control device 100 may change the volume or timbre of the output sound depending on conditions such as the volume, tone, or interval between sound outputs of the performer P1's voice.

[0064] Furthermore, if performer P1 utters a sound that satisfies the first condition, such as "dodo," at short intervals, the performance control device 100 may determine that the previous utterance corresponds to a ghost sound and output a quieter bass drum DS3 sound, or the bass drum DS3 sound played as a ghost sound, instead of the normal bass drum DS3 sound. The performance control device 100 may also output decorative sounds in response to the sounds produced by performer P1.

[0065] Furthermore, for example, the performance control device 100 does not need to output a sound that is identical to the rhythm of the voice uttered by the performer P1. For example, when the performer P1 outputs a single note "do", the performance control device 100 may output a sound that is the result of playing the bass drum DS3 multiple times, such as "do do do".

[0066] Furthermore, for example, the performance control device 100 may output sounds from different sound-producing devices depending on the content of the voice uttered by the performer P1. For example, speakers may be installed not only on the bass drum DS3 but also on the snare drum DS2. If the performer P1 makes a sound such as "don" and the movement of the throat corresponds to the sound "don," the performance control device 100 may output the sound of the bass drum DS3 from the speaker SP installed on the bass drum DS3. On the other hand, if the performer P1 makes a sound such as "tan" and the movement of the throat corresponds to the sound "tan," the performance control device 100 may output the sound of the snare drum DS2 from the speaker installed on the snare drum DS2. In other words, the performance control device 100 may output sounds corresponding to the voice and throat movement from speakers corresponding to the voice and throat movement uttered by the performer P1.

[0067] Furthermore, the output sound database 121 of the performance control device 100 may contain audio data in lossy compressed audio formats such as MP3 (MPEG-1 Audio Layer-3) as sound data. In other words, the performance control device 100 may output sound from the sound generator using data in any audio format. Furthermore, waveform data prepared in advance by any user may be registered and stored as sound data in the output sound database 121.

[0068] Furthermore, the performance control device 100 may not only achieve performance by outputting pre-recorded sounds, but may also output sounds from some of the sound-producing devices by operating a mechanical device that plays some of the sound-producing devices among the multiple sound-producing devices. For example, if the information detected by the microphone MC and the sensor SE each meets predetermined conditions, the performance control device 100 may use a motor or the like to mechanically operate the drum pedal that plays the bass drum DS3, thereby playing the bass drum DS3.

[0069] [5-5. Others] The performance control device 100 described above acquires the sound pickup result from the microphone MC, which detects the sound emitted by the microphone MC of performer P1 who performs using multiple sound-producing instruments, and the movement of the performer's throat detected by the sensor SE. Based on the acquired sound pickup result and the movement of the performer's throat detected by the sensor SE, it outputs the sound emitted by some of the multiple sound-producing instruments played by performer P1. However, as will become clear from the above explanation, the performance control device 100 also has the following features. That is, the performance control device 100 acquires the detection result from the microphone MC, which picks up the sound emitted by the controller (e.g., performer P1) in order to perform control actions (e.g., not only the act of playing sound-producing instruments, but also actions of controlling control objects such as damper pedals) of the controller who controls multiple control objects (e.g., various control objects such as multiple sound-producing instruments or damper pedals), and the detection result from the detection device (e.g., sensor SE), which acquires information of a different type from that of the microphone MC, regarding such control actions. Then, if the sound pickup result from the microphone MC satisfies a predetermined first condition and the detection result from the sensor SE satisfies a predetermined second condition, the performance control device 100 controls some of the multiple control targets. As a result of this processing, the performance control device 100 can improve the degree of control freedom of the operator.

[0070] [5-6. Program] The functions of the playback control device 100 described above may be realized through the cooperation of one or more processors constituting the playback control device 100 and a program stored in a predetermined storage device (e.g., storage unit 120). The program according to this disclosure can be provided in a form stored on a computer-readable recording medium and installed on a computer. The recording medium is, for example, a non-transitory recording medium and may be implemented by an optical recording medium (optical disc) such as a CD-ROM, or by any known type of recording medium such as a semiconductor recording medium or a magnetic recording medium. Note that a non-transitory recording medium includes any recording medium except for transient propagation signals (transitory, propagating signals), and does not exclude volatile recording media. Furthermore, in a configuration in which a distribution device such as a server distributes a program via a communication network, the storage device that stores the program in such a distribution device may correspond to the aforementioned non-transitory recording medium.

[0071] [5-7. Regarding processing methods] Of the processes described in the above embodiments, all or part of the processes described as being performed manually can also be performed automatically by known methods. In addition, the processing procedures, specific names, and various data or parameters shown in the above text and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown.

[0072] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads or usage conditions.

[0073] Furthermore, the embodiments and modifications described above can be combined as appropriate, provided that the processing content is not inconsistent.

[0074] Although some embodiments of the present invention have been described in detail above with reference to the drawings, these are illustrative examples, and the present invention can be implemented in various other forms with modifications and improvements based on the knowledge of those skilled in the art, starting with the embodiments described in the disclosure section of the invention.

[0075] Furthermore, the configuration of the aforementioned performance control device 100 can be flexibly changed, for example, by calling external platforms, etc., via API (Application Programming Interface) or network computing, depending on the function.

[0076] Furthermore, the term "part" in the claims may be replaced with "means" or "circuit," etc. For example, "acquisition part" may be replaced with "acquisition means" or "acquisition circuit." [Explanation of Symbols]

[0077] 1. Performance System 100 Performance control device 110 Communications Department 120 Storage section 121 Output Sound Database 122 Condition Database 130 Control Unit 131 Acquisition Department 132 Judgment section 133 Output section 134 Settings Section

Claims

1. An acquisition unit that acquires the sound emitted by a performer who uses multiple sound-producing instruments to play some of the instruments via a microphone, and acquires the movement of the performer's throat via a predetermined detection device, Based on the sound acquired by the acquisition unit and the movement of the throat, the output unit outputs the sound emitted by some of the multiple sound-producing devices played by the performer. A performance control device characterized by having the following features.

2. The output unit outputs the sound emitted by the aforementioned sound generator when the sound acquired via the microphone is a sound predetermined to correspond to the sound of the aforementioned sound generator. The performance control device according to feature 1.

3. The output unit outputs the sound emitted by the aforementioned sound-producing device when the sound acquired via the microphone satisfies a predetermined first condition and the throat movement detected by the predetermined detection device satisfies a predetermined second condition. The performance control device according to feature 1.

4. The acquisition unit acquires the movement of the throat when the performer makes a sound in order to play the sound-producing device, via the predetermined detection device. The performance control device according to feature 1.

5. The acquisition unit acquires, via the microphone, the sounds made by a performer playing a drum set consisting of multiple percussion instruments, which are multiple sound-producing devices, in order to play some of the percussion instruments, and also acquires the movements of the performer's throat via the predetermined detection device. The output unit outputs a pre-recorded sound that is emitted by some of the percussion instruments, based on the sound acquired by the acquisition unit and the movement of the throat. A performance control device according to any one of features 1 to 4.

6. The acquisition unit acquires, via the microphone, the sounds made by a performer playing a drum set consisting of multiple percussion instruments, which are multiple sound-producing devices, in order to play some of the percussion instruments, and also acquires the movements of the performer's throat via the predetermined detection device. The output unit, based on the sound acquired by the acquisition unit and the movement of the throat, operates a performance device that plays some of the multiple percussion instruments, thereby outputting the sound of those percussion instruments. A performance control device according to any one of features 1 to 4.

7. A performance control method performed by a performance control device, The acquisition process involves acquiring the sound produced by a performer who uses multiple sound-producing instruments to play some of the instruments via a microphone, and acquiring the movement of the performer's throat via a predetermined detection device. Based on the sound acquired in the acquisition step and the movement of the throat, an output step is performed to output sounds emitted by some of the multiple sound-producing devices played by the performer. A performance control method characterized by including the following.

8. One or more musical instruments composed of multiple sound-producing devices, A microphone that captures the sound produced by a performer using the aforementioned musical instrument in order to play some of the sound-producing instruments, A detection device that acquires the movement of the performer's throat, A performance control device that outputs sounds emitted by some of the multiple sound-producing devices played by the performer, based on the sound picked up by the microphone and the throat movements acquired by the detection device. A performance system characterized by having the following features.

Images