Control device

Abstract

According to one embodiment, a control device includes a first transmission unit, a first reception unit, and a first generation unit. The first transmission unit transmits, to the second communication site, first performance data including performance content of the keyboard instrument at the first communication site. The first reception unit receives second performance data from the second communication site. The first generation unit generates a drive signal for sounding in accordance with the second performance data, and outputs the drive signal to the sounding device at the first communication site. At least one of the first performance data and the second performance data includes a key position signal indicating the amount of press of a key in the keyboard instrument.

Description

Technical Field

[0001] The present invention relates to a control device. Background Art

[0002] Technology is being developed to connect multiple communication bases for playing musical instruments via a network, enabling ensemble performances even in separate locations. For example, Patent Document 1 discloses a technology that reduces the effects of communication delays to achieve a comfortable ensemble performance.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: (Japanese) Patent Publication No. 2005-195982 Summary of the Invention

[0006] In a performance performed at multiple communication locations, it is difficult for multiple performers to achieve a sense of unity from various perspectives, compared to a performance performed at the same location.

[0007] One of the objects of the present invention is to enable a plurality of players performing an ensemble to gain a sense of unity.

[0008] Technical solutions to technical problems

[0009] In one embodiment, a control device includes a first transmitting unit, a first receiving unit, and a first generating unit. The first transmitting unit transmits first performance data, including the content of a performance on a keyboard instrument at a first communication base station, to a second communication base station. The first receiving unit receives second performance data from the second communication base station. The first generating unit generates a drive signal for producing sound corresponding to the second performance data and outputs it to a sound producing device at the first communication base station. At least one of the first performance data and the second performance data includes a key position signal indicating the amount of depression of a key on the keyboard instrument.

[0010] Effects of the Invention

[0011] According to the present invention, a plurality of players performing an ensemble can obtain a sense of unity. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is a diagram for explaining the communication system configuration in the first embodiment.

[0013] Figure 2 These are diagrams illustrating the internal structure of the automatic player piano in the first embodiment.

[0014] Figure 3 This is a diagram for explaining the components of the control device in the first embodiment.

[0015] Figure 4 This is a diagram for explaining the components of the ensemble control function in the first embodiment.

[0016] Figure 5 This is a diagram illustrating the positional relationship between the vibrator and the sound pickup sensor in the second embodiment.

[0017] Figure 6 This is a diagram illustrating components of a drive signal generating section in the second embodiment.

[0018] Figure 7 This is a diagram explaining the relationship between the speed and the delay time in the third embodiment.

[0019] Figure 8 This is a diagram explaining the relationship between the speed and the correction value in the third embodiment.

[0020] Figure 9 This is a diagram illustrating components of an environment collection device at a communication base T1 in the fourth embodiment.

[0021] Figure 10 This is a diagram for explaining components of an environment providing device at a communication base T2 in the fourth embodiment.

[0022] Figure 11 This is a diagram for explaining components of a control signal generating unit in the fifth embodiment.

[0023] Figure 12 This is a diagram for explaining a display example of a screen in the sixth embodiment. DETAILED DESCRIPTION

[0024] Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. The embodiment shown below is an example, and the present invention should not be construed as being limited to these embodiments. The components described in each embodiment can also be applied to other embodiments. In the drawings referred to by the multiple embodiments described below, there are cases where the same parts or parts with the same functions are marked with the same figure marks or similar figure marks (only symbols such as A and B are marked after the numbers), and their repeated descriptions are omitted. For the drawings, there are cases where a part of the structure is omitted from the drawings to make the description clear and the description is schematically provided.

[0025] <First embodiment>

[0026] [Communication System]

[0027] Figure 1This is a diagram illustrating the components of a communication system in one embodiment. The communication system includes a server 1000 connected to a network NW such as the Internet. The server 1000 includes a control unit such as a CPU, a storage unit, and a communication unit. The control unit provides a service for achieving ensemble between communication bases by executing a prescribed program. The server 1000 controls the communication between a plurality of communication bases connected to the network NW, and performs the processing required to achieve P2P type communication between the automatic pianos 1 at each communication base. This processing can be achieved by a well-known method. Figure 1 In the example, two communication bases T1 and T2 are shown, but the number is not limited to this number, and there may be more communication bases. In the following description, when the communication bases T1 and T2 are not distinguished from each other, they are simply referred to as communication bases.

[0028] In this example, communication base T1 and communication base T2 exchange information related to performances at each communication base via P2P communication. This communication enables ensembles across multiple communication bases. Each communication base is equipped with an automatic piano 1. In this example, an environment collection device 82 and an environment provision device 88 are connected to the automatic piano 1.

[0029] The environment collection device 82 includes sensors for collecting information about the surrounding environment of the automatic piano 1 and outputs a collection signal representing the sensor's measurement results. Examples of the surrounding environment include sound, light, vibration, temperature, and air flow. When the environment providing device 88 receives a control signal representing the surrounding environment, it provides an environment based on the control signal. The environment collection device 82 and the environment providing device 88 can be integrally formed. The environment providing device 88 can also be provided in accordance with the number of other communication bases. For example, if there are three communication bases other than communication base T1, three environment providing devices 88 can be provided at communication base T1, corresponding to each communication base. At least one of the environment collection device 82 and the environment providing device 88 is a component that can be incorporated into the automatic piano 1. Specific examples of the environment collection device 82 and the environment providing device 88 will be described later.

[0030] The automatic player piano 1 includes a keyboard musical instrument 10 , a control device 20 , a sensor 30 , and a driving device 40 .

[0031] [Player Piano]

[0032] Next, the components of the automatic player piano 1 will be described.

[0033] Figure 2This figure illustrates the internal structure of the automatic piano in the first embodiment. In the automatic piano 1, the keyboard instrument 10 corresponds to, for example, a grand piano. The keyboard instrument 10 includes a plurality of keys 12. The keyboard instrument 10 includes hammers 14, strings 15, and dampers 18, which are arranged corresponding to the keys 12. In the automatic piano 1, the components arranged corresponding to the keys 12 are emphatically shown. Figure 2 The components shown are provided corresponding to one key 2. Therefore, description of the components provided corresponding to other keys 12 is omitted. Some keys 12 may not be provided with a damper 18 or other components.

[0034] The keyboard instrument 10 includes a plurality of pedals 13. The plurality of pedals 13 are, for example, a damper pedal, a soft pedal, and a sostenuto pedal. In the automatic piano 1, the components corresponding to the pedals 13 are shown in FIG. Figure 2 The components are provided corresponding to one pedal 13. Therefore, description of the components provided corresponding to the other pedals 13 is omitted. The keyboard instrument 10 further includes a keyboard cover 11, bridges 16, a soundboard 17, and straight posts 19.

[0035] The sensor 30 includes a key sensor 32, a pedal sensor 33, and a hammer sensor 34. The key sensor 32 is provided corresponding to each key 12 and outputs a measurement signal corresponding to the movement of the key 12 to the control device 20. In this example, the key sensor 32 outputs a measurement signal corresponding to the position (depression amount) of the key 12 to the control device 20. The position of the key 12 can be measured continuously (with fine resolution) or by detecting when the key 12 passes through a predetermined position. The positions of the key 12 can be detected at multiple positions within the pressing range of the key 12 (from the idle position to the end position).

[0036] Hammer sensors 34 are provided corresponding to each hammer 14 and output measurement signals corresponding to the movement of the hammer 14 to the control device 20. In this example, the hammer sensors 34 measure the position (rotation amount) of the hammer shank immediately before the hammer 14 strikes the string 15 and output measurement signals corresponding to the measurement results to the control device 20. The position of the hammer shank can be measured continuously (with fine resolution) or by detecting when the hammer shank passes through a predetermined position. The positions at which the hammer shank is detected can be multiple positions within the range immediately before the hammer 14 strikes the string 15.

[0037] A pedal sensor 33 is provided corresponding to each pedal 13 and outputs a measurement signal corresponding to the movement of the pedal 13 to the control device 20. In this example, the pedal sensor 33 outputs a measurement signal corresponding to the position (depression amount) of the pedal 13 to the control device 20. The position of the pedal 13 can be detected continuously (with fine resolution) or based on the pedal 13 passing through a predetermined position. The positions of the pedal 13 can be detected at multiple positions within the pedal 13's depression range (from the idle position to the end position).

[0038] The drive device 40 includes a key drive device 42, a pedal drive device 43, a limiter 44, an exciter 47, and a damper drive device 48. The key drive device 42 is provided corresponding to each key 12 and is driven by a drive signal from the control device 20 to depress the key 12. This mechanically reproduces the same situation as when a player presses down the key 12. The pedal drive device 43 is provided corresponding to each pedal 13 and is driven by a drive signal from the control device 20 to depress the pedal 13. This mechanically reproduces the same situation as when a player steps on the pedal 13. The damper drive device 48 is provided corresponding to each damper 18 and is driven by a drive signal from the control device 20 to separate the damper 18 from the strings 15. The damper drive device 48 can be configured to drive all dampers 18 simultaneously.

[0039] The stopper 44 is driven under control of the control device 20 so that it is positioned either in a position where it collides with the hammer shank (a blocking position) or in a position where it does not collide with the hammer shank (a retracted position). When the stopper 44 is in the blocking position, even when a key 12 is depressed, the movement of the hammer shank is restricted, and the hammer 14 does not strike the string 15. When the stopper 44 is in the retracted position, the hammer 14, in conjunction with the key 12, strikes the string 15 when the key 12 is depressed. This striking of the string 15 causes the keyboard instrument 10 to produce a sound.

[0040] In this example, the vibrator 47 is supported by a support portion connected to the straight support post 9 so as to contact the surface of the soundboard 17 opposite the portion where the bridge 16 is located. The vibrator 47 vibrates the soundboard 17 under control of a drive signal from the control device 20. For example, when a drive signal containing a piano sound is supplied from the control device 20, the vibrator 47 applies vibrations corresponding to the drive signal to the soundboard 17. As a result, the piano sound is emitted from the soundboard 17. Multiple vibrators 47 may be arranged in contact with the soundboard 17. Instead of the vibrator 47 that vibrates the soundboard 17, a speaker that emits sound may be used.

[0041] The sound produced by the keyboard instrument 10 includes sound produced by striking the strings 15 with the hammers 14 and sound produced by vibrating the soundboard 17 with the vibrator 47. Therefore, the keyboard instrument 10 may also include a sound producing device that produces a struck sound by driving the keys 12, and a sound producing device that produces a sound from the soundboard 17 by driving the vibrator 47. As will be described later, the driving of the keys 12 and the driving of the vibrator 47 are achieved by outputting a driving signal to the driving device 40.

[0042] The components of the control device 20 will be described. In this example, the control device 20 is installed in the keyboard instrument 10. The control device 20 may not be installed in the keyboard instrument 10, and may be, for example, a personal computer, a tablet computer, a mobile phone, or the like.

[0043] Figure 3 This is a diagram illustrating components of the control device in the first embodiment. The control device 20 includes a control unit 21 , a storage unit 22 , an operation panel 23 , a communication unit 24 , a sound source unit 25 , and an interface 26 . These components are connected via a bus 27 .

[0044] The control unit 21 is an example of a computer having a processor such as a CPU and a storage device such as a RAM. The control unit 21 uses the CPU (processor) to execute the program stored in the storage unit 22, and realizes the functions for performing various processing in the control device 20. The functions realized in the control device 20 include the ensemble control function described later. The ensemble control function controls the various parts of the control device 20 and the various components of the interface 26. The sensor 30 and the drive device 40 are connected to the interface 26. In this example, the interface 26 is also connected to an external device 80. The interface 26 sends drive signals, control signals, etc. generated by the control unit 21 to the components that become the object, and receives measurement signals, collection signals, etc. from the various components that become the object.

[0045] The storage unit 22 is a storage device such as a nonvolatile memory or a hard disk drive, and stores a program executed by the control unit 21 and various data required for executing the program.

[0046] The operation panel 23 has operation buttons and the like for accepting user operations. When the operation buttons accept user operations, an operation signal corresponding to the operation is output to the control unit 21. The operation panel 23 may have a display screen. In this case, the operation panel 23 may be a touch panel that combines a touch sensor and a display screen.

[0047] The communication unit 24 is a communication module that communicates with other devices via wireless or wired channels. In this example, the other device with which the communication unit 24 communicates is the server 1000 or an automatic piano at another communication location. In this example, performance data representing the performance content of the keyboard instrument 10, environmental data, and the like are communicated between the communication locations.

[0048] The sound source unit 25 generates a sound signal under the control of the control unit 21. The sound signal is used in a drive signal (excitation drive signal described later) for driving the vibrator 47. In this example, the sound signal includes a signal representing the sound of a piano. The control unit 21 controls the sound source unit 25 to generate a sound signal, which, for example, represents the sound of a piano corresponding to the performance content corresponding to the performance data. The performance data can be data generated based on the measurement signal generated by the sensor 30. The performance data can be data in the form of MIDI containing, for example, sound control information such as note on, note off, note number, speed, etc., or it can be information directly represented by the measurement signal.

[0049] The interface 26 connects the control device 20 to various external components. As described above, the components connected to the interface 26 in this example include the sensor 30, the drive device 40, and the external device 80. The interface 26 outputs the measurement signal output from the sensor 30 to the control unit 21. The interface 26 outputs a drive signal for driving each device to the drive device 40. The drive signal is generated by the ensemble control function 100 described later. The interface 26 may also include a headphone terminal, etc., which supplies a sound signal representing the piano sound generated by the sound source unit 25.

[0050] [Ensemble control function]

[0051] Next, the ensemble control function implemented by the control unit 21 executing a program will be described. The components implementing the ensemble control function are not limited to those implemented by executing a program; at least some of the components may be implemented by hardware. The components implementing the ensemble control function may be implemented not by the control unit 20 but by a device connected to the interface 26 (e.g., a computer installed with the program).

[0052] In this example, when implementing the ensemble control function, the control unit 21 controls the stoppers 44 to the blocking position. In this case, when a user inputs a performance operation to the keys 12 and pedals 13, the stoppers 44 block the string strikes. Meanwhile, the sound source unit 25 generates a sound signal (e.g., a piano note) corresponding to the performance operation. Using this sound signal, the vibrator 47 vibrates the soundboard 17, thereby emitting sound. The signal for driving the vibrator 47 is generated by the drive signal generation unit 145, described below.

[0053] Figure 4 This figure illustrates the components of the ensemble control function in the first embodiment. The ensemble control function 100 includes a performance data generation unit 131, a performance data transmission unit 133, a performance data reception unit 143, and a drive signal generation unit 145. In this example, the ensemble control function 100, as a function further supplementing the ensemble, is configured to share the surrounding environment of the automatic piano 1 between communication bases. The function includes an environment data generation unit 121, an environment data transmission unit 123, an environment data reception unit 183, and a control signal generation unit 185.

[0054] The performance data generation unit 131 generates performance data representing the content of a performance on the keyboard instrument 10 based on the measurement signals output from the sensors 30. In this example, the performance data includes the measurement signals output from the key sensors 32 (hereinafter referred to as key position signals) and the measurement signals output from the pedal sensors 33 (hereinafter referred to as pedal position signals). In this example, the key position signals include the pitch of the depressed key 12 and the amount of depression of the key 12. If the key sensors 32 are sensors that measure the amount of depression of the key 12 at four locations, the information on the amount of depression of the key 12 included in the key position signals indicates one of the four locations.

[0055] In this example, the pedal position signal includes the type of pedal 13 pressed and the amount of pedal 13 depressed. If the pedal sensor 33 measures the amount of pedal depression at three locations, the information about the amount of pedal 13 depressed is displayed at one of the three locations. The performance data may also include a measurement signal output from the hammer sensor 34 (hereinafter referred to as a hammer position signal). The hammer position signal includes, for example, the pitch of the key and the rotational position of the hammer 14.

[0056] It is assumed that the performance data generated by the performance data generating unit 131 is data (e.g., in MIDI format) including sound control information generated based on the measurement results of the key sensor 32 and the pedal sensor 33. In this case, for example, when transmitting a note-on, the depression amount on the key 12 must be advanced to a state where the note-on is generated.

[0057] On the other hand, the performance data generation unit 131 in this example can sequentially transmit the amount of key 12 depressed while the key 12 is in the process of being depressed. Therefore, the automatic pianos 1 at other communication bases can recognize that a key 12 has been depressed before a note-on occurs. For example, when a key 12 is depressed at the automatic piano 1 at communication base T1, the keys 12 at the automatic piano 1 at communication base T2 can be driven to the recognized amount of depression before a note-on occurs. This allows communication base T2 to drive the keys 12 with a short delay, thus following the performance operation on the keys 12 at communication base T1.

[0058] The performance data transmitting unit 133 transmits the performance data generated by the performance data generating unit 131 to other communication locations.

[0059] The performance data receiving unit 143 receives performance data transmitted from another communication base.

[0060] The drive signal generating section 145 generates drive signals for use in the drive device 40 based on the performance data received by the performance data receiving section 143. The drive signals include signals (key drive signals) supplied to the key drive devices 42, signals (pedal drive signals) supplied to the pedal drive device 43, and signals (vibration drive signals) supplied to the vibrator 47.

[0061] The key drive signal is generated based on the performance data, more specifically, based on the key position signal included in the performance data. The key drive signal is a signal used to control the key drive device 42 to drive the key 12 to reproduce the amount of depression corresponding to the key position signal. The key 12 to be driven is the key corresponding to the pitch determined by the key position signal. The pedal drive signal is generated based on the performance data, more specifically, based on the pedal position signal. The pedal drive signal is a signal used to control the pedal drive device 43 to move the pedal corresponding to the type determined by the pedal position signal to a position corresponding to the amount of depression.

[0062] The vibration driving signal is generated based on the performance data, more specifically, based on the key position signal and the pedal position signal, via the sound source unit 25. When the vibrator 47 vibrates the soundboard 17 in response to the vibration driving signal, the sound corresponding to the signal generated by the sound source unit 25 (in this example, a piano sound) is diffused around the keyboard instrument 10 via the soundboard 17.

[0063] When the sound source unit 25 generates a sound signal, the drive signal generating unit 145 can generate sound control information based on the key position signal and the pedal position signal, and cause the sound source unit 25 to generate a sound signal based on the sound control information. In this case, the drive signal generating unit 145 can generate the sound control information using a calculation that predicts the time and velocity of a note-on based on changes in the amount of depression of the key 12 indicated by the key position signal in the performance data. Changes in the rotational position of the hammer 14 indicated by the hammer position signal in the performance data can be used for this prediction calculation. The prediction calculation can use a pre-learned model acquired through machine learning, or it can use a fitting process that assumes a constant velocity trajectory, a constant acceleration trajectory, or the like based on changes in the amount of depression. This improves prediction accuracy even when the movements of the key 12 and the hammer 14 are inconsistent.

[0064] Consider a situation where a key 12 is driven based on performance data, and the hammer 14 that is actuated as a result strikes the string 15. In this case, since time is required from a sounding instruction (e.g., a note-on) to the actuation of the key 12, the timing of the sounding is delayed. Therefore, the timing of the sounding is affected not only by the communication delay between communication base stations but also by the delay in the actuation of the key 12.

[0065] On the other hand, although the key drive signals and pedal drive signals are used by the drive signal generator 145 to drive the keys 12 and pedal 13, the stoppers 44 prevent the hammers 14 from striking the strings 15, so no hammer sound is produced. Instead, the vibrator 47 is driven by the vibration drive signal, and sound is produced from the soundboard 17. Sound production using the vibrator 47 does not require driving the keys 12. Therefore, the time from a sound production instruction (e.g., note-on) to the actual sound production is shorter when sound is produced using the vibrator 47 than when sound is produced using hammer sound.

[0066] At this time, since the sound of the vibrator 47 and the driving of the key 12 are controlled separately, the timings thereof are deviated from each other. On the other hand, since the amount of deviation is very small, the influence on the user's feeling is very small.

[0067] The transmitted performance data and sound production method are not limited to the above-described combinations. For example, rather than transmitting the amount of key 12 depressed as performance data, sound production control information may be transmitted as performance data. In this case, by using the sound production by the vibrator 47 as described above, the time difference in sound production between communication bases can be shortened compared to using sound production by striking the strings.

[0068] Each drive signal is generated based on the sound control information in the performance data. In this case, the velocity value in the sound control information can be increased to a predetermined value or above for the key drive signal. When the velocity value is small, that is, when the key 12 is depressed slowly (when the hammer 14 rotates slowly), the key drive device 42 slows down the drive speed of the key 12.

[0069] At this time, according to the characteristics of the solenoid, there is a situation where the slower the driving speed is set, the more delayed the action of the key 12 is compared to the predetermined moment. In order to compensate for the delay, it is considered to increase the speed value. If the speed value is increased, although the delay time when the key 12 is driven is shortened, the striking sound becomes louder. On the other hand, in this example, since the hammer 14 is prevented from striking the string by the limiter 44 and no striking sound is produced, there is no effect on the pronunciation. Therefore, the speed value can be increased for the key 12 that does not contribute to the pronunciation. The speed value is not changed for the vibration drive signal so that the pronunciation content remains unchanged. At this time, in order not to affect the user's performance, some of the keys 12 can be not driven. For the keys 12 that are not driven, the keys of the pitch used in the performance can be used as targets by pre-setting the performance.

[0070] As another example, instead of using a vibrator to produce sound, the limiter 44 can be controlled in a retracted position, and the sound produced by driving the key 12 and striking the string can be used. In this case, as described above, by transmitting the amount of key 12 depressed as performance data, the time difference in sound production between communication bases can be shortened compared to transmitting sound production control information as performance data. In this case, both the sound produced by the user's performance operation (e.g., the sound produced by performance at communication base T1) and the sound produced by performance at another communication base (e.g., communication base T2) include the striking sound.

[0071] At this time, in order not to affect the user's performance, the pedal 13 may not be driven by the pedal driving device 43 but the dampers 18 may be driven by the damper driving device 48 .

[0072] As another example, the limiter 44 can be controlled to the avoidance position and the vibrator 47 can be used to produce sound. In this case, the drive signal generation unit 145 does not generate key drive signals and pedal drive signals, and the keys 12 and pedals 13 can be kept stationary. In this way, the sound produced by the user's performance operation includes the string sound, while the sound produced by the performance at another communication base station can be the sound produced by the vibrator 47 with less delay. Although multiple examples of the combination of transmitted performance data and sound production methods have been described, the selection of which method can be made by operating the operation panel 23.

[0073] The environmental data generation unit 121 generates environmental data representing the surrounding environment based on the collected signal output from the environment collection device 82. In this example, the surrounding environment includes images and sounds surrounding the device. Therefore, the environment collection device 82 includes a device for collecting the surrounding environment, namely, a camera (imaging device) for capturing images and a microphone (sound pickup device) for capturing sounds. In this example, the camera captures images of the area including the performer of the keyboard instrument 10.

[0074] The image information contained in the environmental data may be image information representing the image (video) itself, and in this example, includes motion information obtained by acquiring the performer's motion using motion capture technology. The sensor for measuring the performer's motion is not limited to a camera, but may also include an IMU (Inertial Measurement Unit), a pressure sensor, a displacement sensor, and the like. The motion information is, for example, information represented by the coordinates of multiple parts having specified features extracted from an image. The environmental data may be sent in the form of audio data representing a sound signal. In this case, the motion information can be sent as data of a channel specified in the audio data, thereby synchronizing the sound signal contained in the audio data. Similarly, the environmental data can be sent to other communication bases after converting the data format, so that it can be sent as part of existing data such as a format representing pronunciation control information (for example, a MIDI format) or a format of video data.

[0075] The sounds collected by the environment collection device 82 may include sounds (piano sounds) produced by playing the keyboard instrument 10. The period during which the sounds produced by playing the keyboard instrument 10 can be determined based on key position signals, etc. If the sounds produced by the performance are produced by the vibrator 47, the sounds can be identified in the sound source unit 25. Therefore, when generating the environment data, the environment data generation unit 121 may perform signal processing to eliminate the sounds generated by the sound source unit 25 from the sounds included in the collected signals.

[0076] When the sound produced by the performance is a hammer sound, the environment data generation unit 121 may also perform signal processing to eliminate the hammer sound component from the sound included in the collected signal. The hammer sound component can be generated by the sound source unit 25 using the key position signal and the pedal drive signal. During the period when the performance of the keyboard instrument 10 produces sound, the environment data generation unit 121 may generate environment data without using the sound included in the collected signal. In this case, the environment collection device 82 may identify the period of performance and not collect sound during that period.

[0077] The environment data transmitting unit 123 transmits the environment data generated by the environment data generating unit 121 to other communication bases.

[0078] The environment data receiving unit 183 receives environment data transmitted from other communication bases.

[0079] The control signal generation unit 185 generates a control signal for use by the environment providing device 88 based on the environment data received by the environment data receiving unit 183. This control signal is used to reproduce the information about the surrounding environment contained in the environment data. In this example, it includes a signal for displaying an image on a display (display device) and a signal for outputting sound from a speaker (sound player). Therefore, the environment providing device 88 includes a display for displaying images and a speaker for outputting sound. The display can be located in a position easily visible to the performer, such as the keyboard cover 11 or a music stand, within the keyboard instrument 10. When displaying images on the keyboard cover 11, a projector that projects the images onto the keyboard cover 11 can be used instead of a display. If multiple communication bases are used as communication targets, an environment providing device 88 can be installed corresponding to each communication base. In this case, the control signal supplied to the environment providing device 88 is generated based on the environment data received from the communication base corresponding to the environment providing device 88.

[0080] The control signal generating unit 185 can use the motion information contained in the environmental data to generate an image that imitates the performer, and generate a signal for displaying the generated image on the display. In this case, an image can be generated that emphasizes a specific part or motion. The specific part can be, for example, the performer's eyes, face, fingers, etc. The specific motion can be, for example, the movement of the eye line, the movement of the face, the movement of the fingers during the performance, etc. The control signal generating unit 185 can use the motion information contained in the environmental data to generate an image such as a curve that numerically represents the performer's motion, and generate a signal for displaying the generated image on the display. The performer can use the displayed information to cooperate with the performance.

[0081] The control signal generating section 185 may generate a signal for displaying on the display an image based on the performance data received by the performance data receiving section 143. The image based on the performance data may include an image showing the performance content included in the performance data, for example, an image showing operated keys and pedals.

[0082] Thus, according to the ensemble control function 100 of the first embodiment, the time difference in sound production between communication bases can be reduced, and each player can closely feel the surrounding environment of the other player. Therefore, it is possible to give the multiple players performing the ensemble a sense of unity.

[0083] <Second embodiment>

[0084] The performance data included in the performance content is not limited to the performance data representing the performance operation on the keys 12, etc. In the second embodiment, an example is described in which the performance data includes a signal representing the vibration of the soundboard 17 that transmits the striking sound produced by the performance. In this example, the vibration of the soundboard 17 is measured by a sound pickup sensor included in the sensor 30.

[0085] Figure 5 This is a diagram illustrating the positional relationship between the vibrator and the sound pickup sensor in the second embodiment. Figure 5 This is a diagram showing the keyboard instrument 10 as viewed from the bottom. Figure 5 As shown, two vibrators 47 (vibrators 47H and 47L) are provided on the soundboard 17. The vibrators 47H and 47L are connected to the portion of the soundboard 17 that exists between the plurality of ribs 17a. The vibrator 47H is provided at a position corresponding to the saddle 16H among the two bridges 16 (bridge 16H (long bridge) and 16L (short bridge)). The vibrator 47L is provided at a position corresponding to the saddle 16L. The bridge 16H is a bridge that supports the strings 15 on the treble side, and the bridge 16L is a bridge that supports the strings 15 on the bass side. The vibrator 47H is supported by a support portion 97H connected to the straight support 19. The vibrator 47L is supported by a support portion 97L connected to the straight support 19.

[0086] The vibrator 47 is not limited to being provided at a position corresponding to the bridge 16 on the soundboard 17, and may be provided at a position separated from the bridge 16 or at a position corresponding to the rib 17a. When provided at a position corresponding to the rib 17a, the vibrator 47 may be provided on the string 15 side of the soundboard 17.

[0087] The sound pickup sensor 37H is attached to the soundboard 17 near the vibrator 47H, measures the vibration of the soundboard 17, and outputs a measurement signal indicating the measurement result. The sound pickup sensor 37L is attached to the soundboard 17 near the vibrator 47L, measures the vibration of the soundboard 17, and outputs a measurement signal indicating the measurement result. Therefore, the performance data transmitted by the performance data transmission unit 133 to other communication bases and the performance data received by the performance data reception unit 143 from other communication bases include the measurement signal PU1 from the sound pickup sensor 37H and the measurement signal PU2 from the sound pickup sensor 37L.

[0088] Figure 6 This figure illustrates the components of the drive signal generation unit in the second embodiment. The drive signal generation unit 145A generates vibration drive signals DS1 and DS2 based on the measurement signals PU1 and PU2 included in the performance data received by the performance data receiving unit 143. The vibration drive signal DS1 is supplied to the vibrator 47H. The vibration drive signal DS2 is supplied to the vibrator 47L.

[0089] The drive signal generation unit 145A includes a crosstalk processing unit 1451, a sound effect imparting unit 1453, and an amplifier 1455. The crosstalk processing unit 1451 applies a predetermined delay and filtering process to the measurement signal PU1 and adds it to the measurement signal PU2. The crosstalk processing unit 1451 applies a predetermined delay and filtering process to the measurement signal PU2 and adds it to the measurement signal PU1. This reduces the crosstalk components contained in the measurement signals PU1 and PU2.

[0090] The sound effect imparting unit 1453 performs signal processing on the measurement signals PU1 and PU2 to impart acoustic effects such as delay, compression, expansion, and equalization. The amplifier 1455 amplifies the measurement signals PU1 and PU2 and outputs vibration driving signals DS1 and DS2 to the vibrators 47H and 47L.

[0091] For example, if the size and shape of the soundboard 17 differs between the keyboard instrument 10 at communication base station T1 and the keyboard instrument 10 at communication base station T2, differences in the vibration mode of the soundboard 17 occur. This results in differences in the sound emitted from each keyboard instrument 10. The parameters used in the signal processing by the crosstalk processing unit 1451 and the sound effect imparting unit 1453 are set to correspond to the different components of the keyboard instrument 10. This reduces differences in sound production caused by differences in shape, etc., even when the keyboard instruments 10 at different communication base stations differ.

[0092] When a musical performance is performed between multiple communication bases, the vibration of the soundboard 17 measured by the sound pickup sensors 37H and 37L includes not only the vibration caused by the struck string sound but also the vibration of the vibrators 47H and 47L based on the vibration excitation drive signals DS1 and DS2. Therefore, before transmitting the performance data to other communication bases, the performance data transmission unit 133 may perform signal processing on the measurement signals PU1 and PU2 included in the performance data to reduce the components of the vibration excitation drive signals DS1 and DS2.

[0093] <Third embodiment>

[0094] As described above, when the velocity value is low, the key driver 42 takes time to drive the key 12, and the movement of the key 12 is delayed compared to when the velocity value is high. As described above, the velocity value can be increased when no hammer sound is produced. However, when hammer sound is produced, if the velocity value is increased too much, the sound content will change significantly. The third embodiment describes an example for minimizing such changes in sound content.

[0095] Figure 7This is a diagram illustrating the relationship between velocity and delay time in the third embodiment. The horizontal axis is the velocity value, which is a value used as an operation parameter in the drive signal generating unit 145 in order to generate a key drive signal. The vertical axis corresponds to the delay time when the key drive device 42 drives the key 12 based on the key drive signal. The velocity value is obtained, for example, from the information contained in the performance data used by the drive signal generating unit 145. More specifically, the velocity value can be obtained from the key position signal or hammer position signal contained in the performance data through calculation. In the case where the performance data contains pronunciation control information, it can be obtained from the velocity value. In this example, the velocity value ranges from "1" to "127".

[0096] like Figure 7 As shown in FIG, if the speed decreases, the delay time increases. If the speed becomes smaller than Vt, the delay time increases rapidly. Therefore, in this example, the drive signal generating unit 145 performs correction so as to increase the value when the speed is smaller than Vt.

[0097] Figure 8 This is a diagram illustrating the relationship between the speed and the correction value in the third embodiment. Figure 8 As shown, when the speed is less than Vt, the correction value obtained by increasing this value is used as a calculation parameter in the drive signal generation unit 145. For example, when the speed changes from "1" to "Vt", the correction value is set to change from "Va" to "Vt". This can reduce the delay time at low speeds. Since the correction value is larger than the value before correction, the sound is louder than expected. On the other hand, the change in sound volume has less impact on the listener than the effect of reducing delay time. To further enhance the delay time reduction effect, the correction value can be set to "Vb" when the speed is "1", which is larger than "Va".

[0098] It is preferable to minimize the delay when performing an ensemble. On the other hand, if the ensemble is not performed and only the pronunciation based on the performance data is listened to, it is preferable that the delay time does not change within the overall input value range compared to reducing the delay. Therefore, in this case, the drive signal generating unit 145 can generate a key drive signal that is delayed in a manner that intentionally postpones the moment of starting to press the key 12 within a range with a large velocity value (a range above a specified value). At this time, it is possible to gradually reduce the time of the postponed moment as the input value becomes smaller from "Vt" to "1".

[0099] <Fourth embodiment>

[0100] In the fourth embodiment, we will describe an example where different instruments are played at different communication bases. Here, communication base T1 is a concert hall capable of hosting orchestral performances. Communication base T2 is a small recording studio, such as a soundproof room. In this example, the orchestral performance takes place at communication base T1, while the piano performance takes place at communication base T2. In other words, the orchestra at communication base T1 lacks a pianist, while the more distant communication base T2 has a pianist.

[0101] The sound of an orchestral performance at communication base T1 is transmitted to communication base T2. A performer at communication base T2 plays automatic piano 1 while listening to the sound received from communication base T1. The user's performance is transmitted as performance data from automatic piano 1 to the automatic piano at communication base T2. Consequently, automatic piano 1 at communication base T2 produces sounds that reproduce the performance at communication base T1. This allows the performer to listen to the piano along with the orchestral performance, even if there is no pianist at communication base T1. In the fourth embodiment, the performer at automatic piano 1 at communication base T2 can convey the sense of presence of the orchestral performance at communication base T1. The components used to achieve this are described in detail below.

[0102] Figure 9 This figure illustrates the components of the environment collection device at communication base station T1 in the fourth embodiment. At communication base station T1, a conductor's podium CS, an automatic piano 1, and chairs 50 are located on stage ST1 of a concert hall. Stage ST1 is equipped with an environment collection device 82 comprising vibration measuring plates 821 and 822 and a microphone 823. The vibration measuring plate 821 is positioned on stage ST1 at the location where the chairs 50 are located. The vibration measuring plate 821 measures vibrations transmitted through the stage ST1 and outputs a collected signal representing the measured vibrations. The vibration measuring plate 822 is positioned on stage ST1 at the location where the automatic piano 1 is located. The vibration measuring plate 822 measures vibrations transmitted through the stage ST1 and outputs a collected signal representing the measured vibrations. In this example, the microphone 823 is positioned near the chairs 50. The microphone 823 collects incoming sound and outputs a collected signal representing the sound.

[0103] Figure 10This figure illustrates the components of the environment providing device at the communication base station T1 in the fourth embodiment. At the communication base station T2, the automatic piano 1 and the chair 50 are set on the stage ST2 of the recording studio. On the stage ST2, an environment providing device 88 including vibration generating plates 881, 882 and a speaker 883 is provided. The vibration generating plate 881 is arranged at the position where the chair 50 is provided on the stage ST2. The vibration generating plate 881 vibrates based on the control signal. The vibration generating plate 882 is arranged at the position where the automatic piano 1 is provided on the stage ST2. The vibration generating plate 882 vibrates based on the control signal. In this example, the speaker 883 is arranged near the chair 50 (near the performer). The speaker 883 emits sound based on the control signal. When generating the control signal used by the vibration generating plates 881, 882 and the speaker 883, signal processing corresponding to the vibration characteristics, signal processing for changing the transfer characteristics, etc. can be implemented.

[0104] When an orchestra is played at communication base station T1, the vibrations accompanying the performance are transmitted via stage ST1 to vibration measuring plates 821 and 822, where the performance sound is collected by microphone 823. The collected signals output from vibration measuring plates 821 and 822 and microphone 823 are transmitted as environmental data to communication base station T2. ​​When automatic piano 1 is played at communication base station T2, the performance content is transmitted as performance data to communication base station T1.

[0105] As a result, at communication base T1, automatic piano 1 is driven and produces sound based on the performance data from communication base T2. In other words, automatic piano 1 at communication base T1 is driven in accordance with the performance of automatic piano 1 at communication base T2. At communication base T2, speaker 883 produces sound based on the environmental data from communication base T1. This sound, collected by microphone 823, corresponds to the sound of the orchestral performance at communication base T1.

[0106] Furthermore, vibration generating plates 881 and 882 are driven and vibrate based on environmental data from communication base T1. The vibrations of vibration generating plate 881 correspond to the vibrations measured by vibration measuring plate 821 at communication base T1. That is, the vibrations transmitted to chair 50 at communication base T1 are also transmitted to chair 50 at communication base T2. The vibrations of vibration generating plate 882 correspond to the vibrations measured by vibration measuring plate 822 at communication base T1. The vibrations of vibration generating plate 882 correspond to the vibrations measured by vibration measuring plate 822 at communication base T1. That is, the vibrations transmitted to automatic piano 1 at communication base T2 are also transmitted to automatic piano 1 at communication base T2. Therefore, the performer at communication base T2 can experience the same sense of presence as if they were performing at communication base T1.

[0107] The vibration measuring plates 821 and 822 and microphone 823 at communication base station T1 also collect piano sound components generated by driving the automatic piano 1. Therefore, signal processing is performed on the path leading to the vibration generating plates 881 and 882 and speaker 883 at communication base station T2 to reduce the piano sound components. Piano sound components can be generated based on the signal used to drive the automatic piano 1 at communication base station T1. Therefore, for example, the environment data generation unit 121 can perform this signal processing when generating the environment data transmitted from communication base station T1. This allows the environment provided by the environment providing device 88 at communication base station T2 to reduce the impact of the performance at communication base station T2.

[0108] <Fifth embodiment>

[0109] In the fifth embodiment, a description will be given of a component that displays its own image (with respect to the image of the performer transmitted from the other communication base) when the environment providing device 88 displays the image of the performer at the other communication base.

[0110] Figure 11 This diagram explains the components of the control signal generation unit in the fifth embodiment. The control signal generation unit 185B in the fifth embodiment includes a self-image acquisition unit 1851 , a remote image acquisition unit 1853 , and an image synthesis unit 1855 .

[0111] The self-image acquisition unit 1851 acquires self-image information related to an image including the performer based on the collected signal output from the environment collection device 82. The remote-image acquisition unit 1853 acquires remote-image information related to an image including the performer collected by the environment collection device 82 at another communication location based on the environment data received by the environment data receiving unit 183. Both the self-image information and the remote-image information are images including the performer and the keyboard portion of the keyboard instrument 10.

[0112] The image synthesis unit 1855 generates a composite image based on the self-image information and the remote image information. The composite image is an image obtained by extracting the image area of ​​the performer contained in the remote image information and overlaying it with the image in the self-image information. At this point, the image synthesis unit 1855 determines the keyboard portion from the images in the self-image information and the remote image information, respectively, and determines the overlapping position of the performer's image in the remote image information so that the keyboard portions coincide with each other. For example, the image synthesis unit 1855 overlaps the image obtained by applying a transformation matrix to the remote image information with the image in the self-image information to maximize the mutual correlation between the keyboard portions. The image synthesis unit 1855 generates a control signal representing the composite image and outputs it to the environment providing device 88. The environment providing device 88 can be a display that displays the composite image on a monitor, or a projector that projects at least a portion of the image in the remote image information onto the keyboard portion. When using a projector for projection, a predetermined transformation matrix corresponding to the position of the keyboard portion can be applied to the remote image information.

[0113] In a two-person piano performance, each player's position relative to the keyboard differs because each player's range of play is different. When two separate communication bases perform together, the players' positions relative to the keyboard remain roughly the same as when playing a single piano. Therefore, the resulting composite image captures two players playing a single piano.

[0114] In this example, the image synthesis unit 1855 determines whether the images of the two performers are in contact with each other, and in the case of contact, causes the area corresponding to the contact portion to emit light, etc., and corrects the synthesized image in a manner that can determine the area. In this case, the illuminated contact portion can be limited to a part (for example, an arm or hand, etc.). The performer can also recognize that the images of the two performers are in contact outside the image. For example, when the images of the two performers are in contact with each other, the seat of the chair used by the performer can be vibrated. In this case, the component that causes the seat of the chair to vibrate is included in the environment providing device 88 and is controlled by a control signal from the control signal generating unit 185B. In this way, even if the two performers are performing at different communication bases, they can experience the actual situation of performing at the same place.

[0115] The remote image acquisition unit 18 may acquire the aforementioned motion information (remote motion information) instead of the remote image information. In this case, the image synthesis unit 1855 may use the motion information to generate an image imitating the performer and synthesize it with the image in the self-image information to generate a synthesized image. When generating the synthesized image, the image synthesis unit 1855 may delay the image in the self-image information and then overlay the image in the remote image information, or may generate a future predicted image for the delayed period based on the image in the remote image information or the remote motion information and overlay it with the image in the self-image information.

[0116] <Sixth embodiment>

[0117] In the sixth embodiment, a case where two communication bases exist in the same room will be described. In this case, two automatic pianos 1 capable of communicating with each other exist in the same room. In this case, a screen displaying images related to the performance contents of each automatic piano 1 can be placed between the two automatic pianos 1.

[0118] Figure 12 This figure illustrates an example display of the screen portion in the sixth embodiment. An automatic piano 1a corresponding to communication base station T1 and an automatic piano 1b corresponding to communication base station T2 are located in the same room. A screen SC onto which an image is projected by a projector PJ is located between automatic pianos 1a and 1b. In this example, screen SC displays an image corresponding to the performance data communicated between automatic pianos 1a and 1b.

[0119] The displayed image is an image of the sound corresponding to the performance content, and in this example, it is a strip image represented by a length corresponding to the length of the note at a position determined by the pitch and the pronunciation time. Figure 12 In the image, the strip image sba is an image showing sounds corresponding to the performance content of the automatic piano 1a, and the strip image sbb is an image showing sounds corresponding to the performance content of the automatic piano 1a. The strip images sba and sbb are displayed in a manner flowing in the communication direction.

[0120] Specifically, when a key 12 is pressed on the automatic piano 1a, a strip image sba corresponding to the sound corresponding to that key 12 is displayed on the screen, moving toward the automatic piano 1b. Here, the sound corresponding to the strip image sba can be emitted by the automatic piano 1b when the strip image sba reaches the automatic piano 1b. In this case, after receiving the performance data corresponding to the strip image sba, the automatic piano 1b can simply delay driving the key 12 until the time arrives. This relationship remains the same even if the automatic piano 1a and the automatic piano 1b are interchangeable.

[0121] Such a projector PJ and screen SC can be said to be an example of the environment providing device 88. In this case, the two automatic player pianos 1a and 1b share the environment providing device 88.

[0122] Modifications

[0123] The present invention is not limited to the above-described embodiments and includes various other variations. For example, the above-described embodiments are described in detail to facilitate understanding of the present invention and are not limited to necessarily including all of the described components. Below, some variations are described. Although the first embodiment is described as an example of a variation, other embodiments may also be applied as examples of variations. It is also possible to combine multiple variations and apply them to each embodiment.

[0124] (1) The drive signal generating unit 145 is not limited to predicting the sound control information such as note-on based on the change in the amount of pressing of the key 12 represented by the key position signal in the performance data, but may also use other information to predict the sound control information. The drive signal generating unit 145 extracts, for example, the movement of the finger toward the key 12 in the image of the performer contained in the environmental data, and estimates the movement of the key 12 after being pressed based on the change in the movement of the finger, thereby predicting the sound control information. The information representing the finger movement also has information related to the pressing of the key 12. Therefore, the image of the finger or the movement of the finger can also be obtained in the sensor 30. In this case, the information representing the finger movement can be sent through the performance data.

[0125] The sensor 30 may include a component that detects contact or proximity with the key 12. In this case, the performance data generation unit 131 generates and transmits performance data based on the detection result, thereby enabling recognition of the start of key 12 pressing at another communication base before the actual start of key 12 pressing. This improves the prediction accuracy of the sound control information.

[0126] These predictions can use a learned model that has been machine-learned to correlate movement history information such as key 12 movements or finger movements with sound control information such as note-on timing and velocity values. The learned model can be generated in a manner that matches each player.

[0127] Predicting such sound control information is not limited to controlling automatic piano 1 at other communication locations; it can also be used for various linkages. For example, it can be applied to components where a keyboard device and a sound source device are wirelessly connected. For example, if a note-on is generated by pressing a key on the keyboard device and then transmitted to the sound source device, the sound generation timing will be delayed due to communication delays. On the other hand, by transmitting the key movement to the sound source device before the note-on is generated by the keyboard device, the effects of communication delays can be reduced by using predictions of the key movement.

[0128] (2) The key position signal generated in response to the pressing of the key 12 of the automatic piano 1 can be used to control other keys 12 in the same automatic piano 1. For example, control can be performed in response to the pressing of the key 12 so that the key 12 corresponding to the high octave of the key 12 is linked. The linked key 12 is not limited to the high octave, but can also be a predetermined tone. The predetermined tone can be determined relatively with respect to the pitch of the pressed key 12, or can be determined absolutely regardless of the pitch. At this time, by using the key position signal instead of the pronunciation control information, the time difference between the pressing of the key 12 of the performance object and the driving of the linked key 12 can be reduced.

[0129] (3) The control device 20 can record the performance content of the automatic piano 1. The recorded data may be data based on sound production control information or data corresponding to signals output from the sensor 30, such as key position signals, hammer position signals, and pedal position signals.

[0130] (4) The performance data transmitting unit 133 may include information for setting, for each key 12, whether or not to drive a key 12 of the automatic piano 1 at another communication base station, in the performance data and transmit the information. Whether or not to drive may be set by the performer at the communication base station on the transmitting side during the performance, or may be predetermined for a specific key or range. At the communication base station on the receiving side, the automatic piano 1 does not drive the key 12, but instead drives the vibrator 47, in response to the key position signal associated with the key 12 set not to be driven.

[0131] (5) The environment collection device 82 may include a sensor installed on the performer, for example, a sensor for measuring the performer's breathing. The automatic piano 1 can send the performer's breathing measurement results as environmental data to other communication bases, and display information corresponding to the breathing changes on the display of the environment providing device 88 at the other communication bases. The performer's breathing is closely related to the performance action. For example, immediately before starting to play, the performer will often take a deep breath. Therefore, when a deep breath is detected, information indicating this situation or the time from the start of the performance can be displayed on the display. It is assumed that the time from taking a deep breath to starting to play is different depending on the performer, and the time can be set to be different according to the performer. In the prediction of this time, a learned model that has been machine-learned on the correlation between the moment of taking a deep breath and the time to the start of the performance can be used.

[0132] (6) The environment providing device 88 may be a small movable device that can provide various environments, for example, a humanoid shape that imitates some characters. For example, the environment providing device 88 may be a humanoid robot that moves its arms and hands based on a control signal. The environment providing device 88 may also be in a shape that can be worn by the performer (watch type, shawl type, neck type, etc.). The components that provide various environments may be the above-mentioned displays and speakers, for example, heat sources, cooling sources, fans, etc. for controlling temperature, or lighting equipment, projectors, etc. for controlling the brightness, color tone, pattern, etc. of the room. The environment providing device 88 may include, for example, a structure such as a robotic arm for changing the position of a heat source, etc., or may be configured with multiple heat sources so that one of them is driven to substantially change the position of the heat source. The heat source may be used to reproduce the position of the performer at another communication base, for example. The environment collecting device 82 may include sensors corresponding to the environment providing device 88, for example, a temperature sensor, an air volume sensor, an illuminance sensor, etc.

[0133] (7) The environment providing device 88 can both localize the sound and image and reproduce the specified sound field by including a plurality of speakers. In this case, the specified reverberation processing or filtering processing such as FIR can be added to the sound signal included in the environmental data. Information for reproducing the sound field characteristics of the room can be collected in the environment collection device 82 and sent to other communication bases as environmental data. Thus, the environment providing device 88 at the communication base on the receiving side can reproduce the sound field of the room at the communication base on the sending side based on the information included in the environmental data. In this case, the sound field of the room at the communication base on the sending side can be reproduced more accurately by including signal processing for eliminating the sound field characteristics of the room at the communication base on the receiving side. The processing for reproducing such a sound field can be added to the vibration drive signal.

[0134] (8) A common metronome can be synchronized at each communication base station through sound, light, vibration, etc. In the case of absolute time synchronization, for example, time information used in a satellite positioning system such as a GPS signal can be used, or time synchronization technology based on the NTP (Network Time Protocol) can be used. In this case, the BPM value is set, and the start time of the beat can be determined based on the time information. The BPM value can be determined based on a pre-set performance music or can be set by the performer.

[0135] Instead of using absolute time as a benchmark, any communication base station among multiple communication base stations can be used as a benchmark for the metronome. In this case, the beat position can be analyzed based on the performance content at the communication base station that serves as the benchmark and used as a metronome. When each beat position is detected from the performance content at multiple communication base stations, the beat position that is consistent at the largest number of communication base stations can be used as a metronome at other communication base stations. Specified data (data containing pronunciation control information, sound data, video data, etc.) can be regenerated according to these metronomes. Specified data can be obtained by recording the performance. For example, the rhythm pattern of a drum can be regenerated by the setting of the metronome.

[0136] In the case of a metronome implemented by vibration, the player can be made to recognize the beat of the metronome by vibrating a movable component of the automatic piano 1. For example, the drive signal generating unit 145 can generate a signal so that the pedal 13 moves slightly for each beat of the metronome. In the case of a damper pedal, the amount of movement of the pedal 13 is so small that the dampers 18 do not separate from the strings 15. The component that moves for each beat of the metronome is not limited to the pedal 13 and can be a key 12. In this case, it is preferred that the key 12 is pressed down enough so that the string does not strike or the sound of an open note is not produced.

[0137] (9) The transmitted performance data may include information about the time at which the performance data was transmitted. By adjusting the performance data received from multiple communication bases on the time axis in accordance with the time information, it is possible to correct the communication delay by eliminating the time axis deviation. For example, if the automatic piano 1 is not being played, or if it is being played but the performance data is not being transmitted to other communication bases, even if the performance data from multiple other communication bases are received at different times due to communication delay, the automatic piano 1 can be driven with the same delay by shifting the performance data on the time axis so that the time information is consistent.

[0138] By using this time information, the delay time of the performance data received from each communication base can be identified. The drive signal generating unit 145 can generate a drive signal in a manner such that the longer the delay time, the smaller the speed value. The drive signal generating unit 145 can process in a manner such that the longer the delay time, the more resonant sound is added. In this way, it is possible to achieve a sound in the automatic piano 1 that has the effect of indicating the size of the distance by the length of the delay time. That is, it is possible to give the audience the feeling that the sound with a long delay time is being played at a farther place. For each communication base, an image that visually indicates the size of the delay time can be displayed on the display. For each communication base, AR (Augmented Reality) can be used to prompt an image that visually indicates the size of the delay time. For example, the delay time can be converted into a position and distance relationship in the AR space, thereby prompting an image related to each communication base.

[0139] (10) The control device 20 can compare the performance data between multiple communication bases and calculate the correlation, so that the correlation is displayed on the display. The correlation can be calculated using signal processing or DNN (Deep Neural Network), for example. In this case, as described above, the performance data adjusted to the same time information between multiple communication bases can be used to calculate the correlation. The control device 20 can analyze the received performance data to determine the chord or the beat position, and display the determined information on the display. In this case, the chord and beat position with the highest similarity in the performance data at multiple communication bases can be displayed on the display. Light can be irradiated on the keyboard so that the key 12 corresponding to the constituent tone of the chord can be recognized by the player.

[0140] (11) The control device 20 analyzes the chord based on the received performance data and, if the similarity of the chord is higher than a predetermined value, determines it as the current chord. When the oscillator is producing sound in response to the performance operation on the key 12, the control device 20 controls the performance operation on the key 12 other than the note corresponding to the chord and does not drive the oscillator 47.

[0141] (12) The control device 20 analyzes the beat position based on the received performance data, and if the similarity of the beat position is higher than a predetermined value, determines it as the current beat position. When the control device 20 performs a sounding operation through the vibrator 47 based on the performance operation of the key 12, if the key 12 is pressed within a predetermined time range before reaching the next predicted beat position, the control device 20 delays the predicted beat position to achieve the sounding operation through the vibrator 47. In this way, the performance sound can be matched with the beat position.

[0142] (13) The control device 20 determines the volume based on the received performance data and determines the volume of the performance of the automatic piano itself. The volume is determined, for example, by the average value of the speed over a specified period of time in the past. The drive signal generating unit 145 adjusts the volume of the received performance data so that it is close to the volume achieved by the performance itself, thereby generating a key drive signal or a vibration drive signal. When adjusting the volume, it is not necessary to change it suddenly, but to change it gradually. In this way, the volume balance in the ensemble can be adjusted. The volume balance can be set in advance so that the volume of one side is relatively increased.

[0143] (14) When the sound produced by the vibrator 47 is produced in response to a performance operation on a key 12, the control device 20 can delay the time of the sound produced in response to the pressing of the key 12. In this case, the performance data transmitted to and from other communication bases is not delayed. Thus, the performer can consider the delay time and perform in such a way as to press the key 12 in advance, thereby reducing the influence of communication delay on the ensemble.

[0144] (15) In the case where the automatic piano 1 is not installed at a certain communication base, but an acoustic piano is not equipped with the sensor 30 and the drive device 40, the control device 20 may not include components related to the sensor 30 and the drive device 40, but may be composed of a desktop computer, a tablet computer, etc.

[0145] In this case, the control device 20 can convert the performance sounds into performance data and transmit it to other communication bases. The performance sounds can be collected by a microphone, and the control device 20 can analyze the component sounds contained in the collected performance sounds and convert them into sound control information to generate performance data. This processing can also be applied to instruments other than pianos.

[0146] (16) The environment collection device 82 may also include a sensor for detecting the opening and closing of the keyboard cover 11, a sensor for detecting whether the performer is sitting on a chair, or the like. In this case, the environment providing device 88 may include a display that shows whether the keyboard cover 11 is open or closed, or whether the performer is sitting on a chair. The environment providing device 88 may also include a structure that opens and closes the keyboard cover 11 in response to a control signal. In this case, the opening and closing of the keyboard cover 11 at a specific communication base station may be linked to the keyboard covers 11 at other communication base stations.

[0147] (17) The keyboard instrument 10 in the automatic piano 1 is not limited to an acoustic piano such as a grand piano, but may also be an electronic keyboard instrument. The electronic keyboard instrument may be a keyboard device having a structure equivalent to the keys 12, or may be a keyboard device having a membrane structure in which the keys 12 have a membrane structure. In the case of a keyboard device having a membrane structure, since it can be placed on the floor and played by stepping on it with the feet, it can be played even when the hands cannot be used. In the case of a keyboard device that is played by stepping on it with the feet, the playable range is narrow. In such a case, a plurality of pre-set keyboard devices with different ranges can be used to perform by multiple people. In the case of a keyboard device having a membrane structure, it can be arranged on the back of the side table of the machine base. In this case, a rotating mechanism can be provided in the support member supporting the side table to switch so that one of the surface and the back of the side table faces upward.

[0148] (18) At least part of the functions of the control device 20 may be prepared as a plug-in in the software that implements the video conference system.

[0149] (19) The network NW connecting the communication bases can be realized by optical cables or the like.

[0150] (20) The environment collecting device 82 and the environment providing device 88 may include components for being detachably mounted on the automatic player piano 1. Alternatively, the environment collecting device 82 and the environment providing device 88 may include components for installing the environment collecting device 82 and the environment providing device 88 in the automatic player piano 1. In this case, the environment collecting device 82 or the environment providing device 88 may be connected to the interface 26 by being mounted on the automatic player piano 1.

[0151] The above is the description of the modified example.

[0152] As described above, according to one embodiment of the present invention, a control device is provided, comprising: a first sending unit that sends first performance data including performance content of a keyboard instrument at a first communication base to a second communication base; a first receiving unit that receives second performance data from the second communication base; and a first generating unit that generates a drive signal for performing a sound corresponding to the second performance data and outputs the signal to a sound producing device at the first communication base, wherein at least one of the first performance data and the second performance data includes a key position signal indicating the amount of depression of a key on the keyboard instrument.

[0153] The sound producing device may include a vibrator connected to a soundboard of the keyboard instrument. The sound producing device may be configured to produce the sound corresponding to the second performance data by vibrating the vibrator in response to the driving signal.

[0154] The sound producing device may include a key of the keyboard instrument, a hammer that moves in conjunction with the key, and a string that strikes the hammer. The sound produced in accordance with the second performance data may be produced by driving the key in accordance with the drive signal. The drive signal may be a signal for driving the key to reproduce the amount of depression in accordance with the key position signal.

[0155] It may further include: a second sending unit, which obtains first environmental data corresponding to information about the surrounding environment collected by the environment collection device at the first communication base and sends it to the second communication base; a second receiving unit, which receives second environmental data from the second communication base; and a second generating unit, which generates a control signal for providing a surrounding environment corresponding to the second environmental data and outputs it to the environment providing device at the first communication base.

[0156] Description of Reference Numerals

[0157] 1, 1a, 1b: Automatic piano, 10: Keyboard instrument, 11: Keyboard cover, 12: Key, 13: Pedal, 14: Hammer, 15: String, 16, 16H, 16L: Bridge, 17: Soundboard, 17a: Rib, 18: Damper, 19: Straight support, 20: Control device, 21: Control unit, 22: Storage unit, 23: Operation panel, 24: Communication unit, 25: Sound source unit, 26: Interface, 27: Bus, 30: Sensor, 32: Key sensor, 33: Pedal sensor, 34: Hammer sensor, 37H, 37L: Pickup sensor, 40: Drive unit, 42: Key drive unit, 43: Pedal drive unit, 44: Limiter, 47, 47H, 47L: Vibrator, 48: Damper drive unit, 49H, 49L: Support Supporting part, 50: Chair, 82: Environment collecting device, 88: Environment providing device, 100: Ensemble control function, 121: Environment data generating part, 123: Environment data sending part, 131: Performance data generating part, 133: Performance data sending part, 143: Performance data receiving part, 145, 145A: Drive signal generating part, 183: Environment data receiving part, 185, 185B: Control signal generating part, 821, 822: Vibration measuring plate, 823: Microphone, 881, 882: Vibration generating plate, 883: Speaker, 1000: Server, 1451: Crosstalk processing part, 1453: Sound effect imparting part, 1455: Amplifying part, 1851: Self-image acquiring part, 1853: Remote image acquiring part, 1855: Image synthesizing part.

Claims

1. A control device, characterized in that: Include: A first transmission unit that transmits first performance data including performance contents of the keyboard instrument at the first communication base to the second communication base; a first receiving unit configured to receive second performance data from the second communication base station; a first generating unit that generates a driving signal for producing sound corresponding to the second performance data and outputs the driving signal to a sound producing device at the first communication base station; At least one of the first performance data and the second performance data includes a key position signal indicating an amount of depression of a key in the keyboard instrument.

2. The control device according to claim 1, characterized in that The sound generating device comprises a vibrator connected to the soundboard of the keyboard instrument. The sound produced according to the second performance data is generated by the vibration of the vibrator according to the drive signal.

3. The control device according to claim 1, characterized in that: The sound producing device includes keys of the keyboard instrument, hammers linked to the keys, and strings for striking the hammers. The sound produced according to the second performance data is produced by driving the key according to the driving signal, The driving signal is a signal for driving the key in such a manner as to reproduce the depressed amount corresponding to the key position signal.

4. The control device according to any one of claims 1 to 3, characterized in that: Further including: a second sending unit that acquires first environment data corresponding to information about the surrounding environment collected by the environment collecting device at the first communication base and sends the data to the second communication base; a second receiving unit configured to receive second environment data from the second communication base; The second generating unit generates a control signal for providing an ambient environment corresponding to the second environment data, and outputs the control signal to an environment providing device at the first communication base.

Images