Information processing system, information processing method, and program

Abstract

This information processing system includes: a characteristic acquisition unit 51 for acquiring an operation characteristic Xa related to a keyboard instrument; a performance analysis unit 52 for specifying a performance characteristic F corresponding to the operation characteristic Xa by analyzing a result of performance by a user using the keyboard instrument; and a display control unit 53 for displaying the operation characteristic Xa and the performance characteristic F on a display device 34.

Description

Information processing system, information processing method, and program

[0001] The present disclosure relates to a technique for analyzing a musical instrument performance by a performer.

[0002] Techniques for displaying various parameters relating to performance devices such as keyboard instruments have been proposed. For example, Patent Document 1 discloses a technique for displaying adjustment parameters such as pitch for each control of an electronic musical instrument.

[0003] Publication No. 2018-159769

[0004] There is a demand for understanding the relationship between the operating characteristics of a musical instrument and the characteristics of one's own performance (hereinafter referred to as "performance characteristics"). For example, if a user could understand the relationship between the range of volume that a musical instrument can produce (dynamic range) and the range of volume that is actually being played, this could be useful in identifying problems that arise during performance and ultimately improving one's performance technique. In consideration of the above, one aspect of the present disclosure aims to enable a user to easily understand the performance characteristics that are being used to perform a particular musical instrument.

[0005] In order to solve the above problems, an information processing system according to one aspect of the present disclosure includes a motion characteristic acquisition unit that acquires a first motion characteristic related to a first performance device, a performance analysis unit that identifies a first performance characteristic corresponding to the first motion characteristic by analyzing the results of a performance by a user using the first performance device, and a display control unit that displays the first motion characteristic and the first performance characteristic on a display device.

[0006] An information processing method according to one aspect of the present disclosure acquires a first operating characteristic related to a first performance device, analyzes the results of a user's performance using the first performance device, identifies a first performance characteristic corresponding to the first operating characteristic, and displays the first operating characteristic and the first performance characteristic on a display device.

[0007] A program according to one aspect of the present disclosure causes a computer system to function as: a motion characteristic acquisition unit that acquires a first motion characteristic related to a first performance device; a performance analysis unit that identifies a first performance characteristic corresponding to the first motion characteristic by analyzing the results of a performance performed by a user using the first performance device; and a display control unit that displays the first motion characteristic and the first performance characteristic on a display device.

[0008] 1 is a block diagram illustrating the configuration of an information processing system in a first embodiment. FIG. 1 is a block diagram illustrating the configuration of a terminal device. FIG. 1 is a block diagram illustrating the configuration of a keyboard instrument. FIG. 2 is a schematic diagram of a keyboard video. FIG. 3 is a schematic diagram of a performance video. FIG. 4 is a schematic diagram of a performance video. FIG. 5 is a schematic diagram of a pedal video. FIG. 6 is a schematic diagram of a setting screen. FIG. 7 is a schematic diagram of performance data. FIG. 8 is a block diagram illustrating the functional configuration of a terminal device. FIG. 9 is a schematic diagram of a reference screen G1. FIG. 10 is a schematic diagram of a reference screen G2. FIG. 11 is a flowchart of performance analysis processing. FIG. 12 is a schematic diagram of a performance history screen. FIG. 13 is a schematic diagram of a reference screen G3 in a second embodiment. FIG. 14 is a block diagram illustrating the configuration of an information processing system in a third embodiment. FIG. 15 is a schematic diagram of a reference screen G4 in a third embodiment. FIG. 16 is a schematic diagram of a reference screen G5 in a fourth embodiment. FIG. 17 is a schematic diagram of a reference screen G6 in a fifth embodiment. FIG. 18 is a schematic diagram of a reference screen G7 in a sixth embodiment. FIG. 19 is an explanatory diagram of an edited video in a seventh embodiment. FIG. 20 is a block diagram illustrating the functional configuration of a terminal device in a seventh embodiment. FIG. 21 is a schematic diagram of a reference screen G8 in a modified example. It is a schematic diagram of a reference screen G9 in a modified example.It is a schematic diagram of a reference screen G9 in a modified example.It is a graph showing the relationship between key depth and key depression speed.It is a schematic diagram of a reference screen G4 in a modified example.It is a schematic diagram of a reference screen G5 in a modified example.It is a schematic diagram of a reference screen G6 in a modified example.It is a schematic diagram of a reference screen G7 in a modified example.It is a schematic diagram of a reference screen G8 ...

[0009] A: First Embodiment Fig. 1 is a block diagram illustrating the configuration of an information processing system 100 according to the first embodiment. The information processing system 100 is a computer system (i.e., a performance recording system or a performance management system) for recording and managing a performance of a keyboard instrument 10 by a performer Ua. The information processing system 100 includes the keyboard instrument 10, a recording system 20, multiple terminal devices 30 (30a, 30b, 30c), and a measurement system 40.

[0010] Each terminal device 30 is an information device such as a smartphone, a tablet terminal, or a personal computer. Of the multiple terminal devices 30, terminal device 30a is used by a performer Ua who plays the keyboard instrument 10. The performer Ua is, for example, a person who practices playing the keyboard instrument 10. The performer Ua is, for example, a student belonging to a music school. The terminal device 30a is an example of a "control system."

[0011] Of the multiple terminal devices 30, terminal device 30b is used by instructor Ub. Instructor Ub instructs performer Ua in a music classroom. That is, instructor Ub uses terminal device 30 to manage the performer Ua's performance of the keyboard instrument 10. Terminal device 30c is used by guardian Uc (e.g., parent) of performer Ua. Like instructor Ub, guardian Uc uses terminal device 30c to manage the performer Ua's performance of the keyboard instrument 10. In the following description, when there is no need to particularly distinguish between performer Ua, instructor Ub, and guardian Uc, they will be collectively referred to as "user U."

[0012] 2 is a block diagram illustrating the configuration of a terminal device 30 (30a, 30b, 30c). The terminal device 30 includes a control device 31, a storage device 32, a communication device 33, a display device 34, an operation device 35, a sound emitting device 36, and a sound collecting device 37. The terminal device 30 may be realized as a single device, or may be realized as a plurality of devices configured separately from each other.

[0013] The control device 31 is configured with one or more processors that control each element of the terminal device 30. For example, the control device 31 is configured with one or more types of processors such as a central processing unit (CPU), a graphics processing unit (GPU), a sound processing unit (SPU), a digital signal processor (DSP), a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC).

[0014] The storage device 32 is one or more memories that store programs executed by the control device 31 and various data used by the control device 31. The storage device 32 is configured with a known storage medium such as a magnetic storage medium or a semiconductor storage medium. The storage device 32 may be configured with a combination of multiple types of storage media. Furthermore, a portable storage medium that is detachable from the terminal device 30, or a storage medium to which the control device 31 can write or read via a communication network (e.g., cloud storage) may be used as the storage device 32.

[0015] The communication device 33 communicates with peripheral devices (e.g., the keyboard instrument 10, the recording system 20, other terminal devices 30, and the measurement system 40). The communication device 33 communicates with the peripheral devices via wireless communication such as Wi-Fi (registered trademark) or Bluetooth (registered trademark). The communication between the communication device 33 and the peripheral devices may be wired communication. The communication between the communication device 33 and the peripheral devices may be communication via a communication network such as the Internet.

[0016] The display device 34 displays an image under the control of the control device 31. The display device 34 is configured with a display panel such as a liquid crystal panel or an organic EL panel. The operation device 35 is an input device that accepts operations by the users U (Ua, Ub, Uc). For example, an operator operated by the users U or a touch panel that detects contact by the users U is used as the operation device 35. Note that the display device 34 or the operation device 35, which are separate from the terminal device 30, may be connected to the terminal device 30 by wire or wirelessly.

[0017] The sound emitting device 36 emits sound waves under the control of the control device 31. The sound emitting device 36 is an output device such as a speaker or headphones. The sound emitting device 36 may include a D / A converter that converts the acoustic signal of the sound emitting target from digital to analog, and an amplifier that amplifies the acoustic signal. Note that the sound emitting device 36, which is separate from the terminal device 30, may be connected to the terminal device 30 by wire or wirelessly.

[0018] The sound collection device 37 collects ambient sounds to generate an acoustic signal (hereinafter referred to as a "collected sound signal"). Specifically, the sound collection device 37 collects the performance sound emitted from the keyboard instrument 10 by the performer Ua. In other words, the collected sound signal is a signal representing the waveform of the performance sound of the keyboard instrument 10. For example, the sound collection device 37 may be a single microphone or a microphone array in which multiple microphones are arranged. For convenience, an A / D converter that converts the collected sound signal from analog to digital and an amplifier that amplifies the collected sound signal are not shown in the figure. The sound collection device 37 may be separate from the terminal device 30 and connected to the terminal device 30 by wire or wirelessly. The control device 31 stores the collected sound signal generated by the sound collection device 37 in the storage device 32.

[0019] The terminal device 30 can establish communication with the keyboard instrument 10 by logging in to the keyboard instrument 10. Logging in to the keyboard instrument 10 requires authentication processing using, for example, the identification information of the keyboard instrument 10, the identification information (user ID) of the user U, and a password. The terminal device 30 can obtain the identification information by, for example, reading an image code attached to the keyboard instrument 10. The image code is a code represented by an optically readable image (for example, a QR code (registered trademark)). The identification information of the keyboard instrument 10 may be input by the user U, for example, by operating the operation device 35.

[0020] 3 is a block diagram illustrating the configuration of a keyboard instrument 10. The keyboard instrument 10 is an instrument that accepts a performance by a performer Ua, and is equipped with a control device 11, a storage device 12, a communication device 13, a display device 14, an operation device 15, a sound source device 16, a sound output device 17, and a performance mechanism 18. The keyboard instrument 10 is an example of a performance device used by the performer Ua for performance. Note that some or all of the functions of the terminal device 30 may be installed in the keyboard instrument 10.

[0021] The performance mechanism 18 includes a keyboard 181, a sound generation mechanism 183, pedals 184, and a detection device 185. The keyboard 181 is a mechanism with an array of multiple keys 182, including white keys and black keys. Each of the multiple keys 182 is a performance operator that accepts operation (e.g., key depression and key release) by the performer Ua. The sound generation mechanism 183 is an action mechanism that generates performance sounds by vibrating strings in response to operation of each key 182 on the keyboard 181. Specifically, the sound generation mechanism 183 includes strings, which are an example of a sound source, hammers that rotate to strike the strings, and a transmission mechanism (e.g., a wippen, jack, or repetition lever) that rotates the hammer in response to the displacement of each key 182. The sound generation mechanism 183 also includes a mute mechanism that prevents the strings from being struck in mute mode.

[0022] The pedal 184 is a performance operator operated by the performer Ua with the foot. For example, the pedal 184 may be a damper pedal (sustain pedal) for instructing the extension of a performance sound, or a soft pedal for reducing the volume of a performance sound.

[0023] The detection device 185 is a sensor that detects the operation of each key 182 and pedal 184. Specifically, the detection device 185 detects the amount of displacement of each key 182 and pedal 184. The detection device 185 is configured, for example, by an optical, mechanical, or magnetic sensor.

[0024] The control device 11 is composed of one or more processors that control each element of the keyboard instrument 10. For example, the control device 11 is composed of one or more types of processors such as a CPU, a GPU, an SPU, a DSP, an FPGA, or an ASIC.

[0025] The control device 11 generates music data in accordance with the detection result of the detection device 185. The music data is data representing a musical piece performed by the performer Ua. The musical piece to be performed includes a right-hand part that the performer Ua should play with his / her right hand HR and a left-hand part that the performer Ua should play with his / her left hand HL.

[0026] Specifically, the music data is a time series of event data conforming to the MIDI (Musical Instrument Digital Interface) standard, for example. That is, the music data specifies, in time series, the pitches (note numbers) corresponding to the keys 182 operated by the performer Ua and the intensity (velocity) of the operation. The music data also represents the operation of the pedal 184 by the performer Ua. As described above, the control device functions as a music data generator that generates music data in accordance with the results of detection by the detection device 185.

[0027] The display device 14 displays images under the control of the control device 31. The display device 14 is configured with a display panel such as a liquid crystal panel or an organic EL (Electroluminescence) panel. The operation device 15 is an input device that accepts operations by the performer Ua. For example, an operator operated by the performer Ua or a touch panel that detects contact by the performer Ua is used as the operation device 15. Note that the display device 14 or the operation device 15, which are separate from the keyboard instrument 10, may be connected to the keyboard instrument 10 by wire or wirelessly.

[0028] The sound source device 16 generates an audio signal representing a performance sound according to the result of detection by the detection device 185. Specifically, the sound source device 16 generates an audio signal according to the performance data generated by the control device 11. For example, an audio signal is generated having a pitch corresponding to the key 182 whose displacement has been detected by the detection device 185. Note that the control device 11 may realize the function of the sound source device 16 by executing a program stored in the storage device 12. As described above, the keyboard instrument 10 is an electronic musical instrument that can selectively generate natural sounds using the sound generation mechanism 183 and electronic sounds using the sound source device 16. Note that one of the sound generation mechanism 183 and the sound source device 16 may be omitted.

[0029] The sound emitting device 17 emits sound waves under the control of the control device 11. The sound emitting device 17 is an output device such as a speaker or headphones. Specifically, the sound emitting device 17 emits performance sounds represented by acoustic signals generated by the sound source device 16. As described above, the keyboard instrument 10 of the first embodiment can selectively generate performance sounds using the sound generation mechanism 183 and the sound source device 16. The sound emitting device 17 may include a D / A converter that converts the acoustic signals from digital to analog and an amplifier that amplifies the acoustic signals. The sound emitting device 17 may be separate from the keyboard instrument 10 and connected to the keyboard instrument 10 by wire or wirelessly.

[0030] The storage device 12 is one or more memories that store the programs executed by the control device 11 and various data used by the control device 11. The storage device 12 is configured with a known storage medium, such as a magnetic storage medium or a semiconductor storage medium. The storage device 12 may also be configured with a combination of multiple types of storage medium. The storage device 12 may also be a portable storage medium that is detachable from the keyboard instrument 10, or a storage medium to which the control device 11 can write or read via a communication network (e.g., cloud storage). The storage device 12 stores the operating characteristic Xa, tuning information Xb, and history information Xc.

[0031] The operation characteristics Xa are device-specific characteristics (device profile) related to the operation of the keyboard instrument 10. The operation characteristics Xa include a dynamic range, a velocity range, a frequency range, a repeated key characteristic, and a key depression sensitivity.

[0032] The dynamic range is the range of volume (the ratio between maximum and minimum volume) of the performance sounds that the keyboard instrument 10 can generate via the sound generation mechanism 183 or the sound source device 16. The velocity range is the range of key pressing speed (key pressing strength). Because the volume of the performance sounds generated by the sound source device 16 corresponds to the key pressing speed, the velocity range is also expressed as the range of performance sound volume. The frequency range is the range of frequencies of the performance sounds that the keyboard instrument 10 can generate via the sound generation mechanism 183 or the sound source device 16. The repeated keying characteristic is the maximum number of key presses per unit time. The key pressing sensitivity is the relationship between the key pressing speed or displacement amount and the performance volume (touch sensitivity) for each key 182. Key pressing sensitivity includes, for example, key depth (mm), half-touch depth (mm), aftertouch depth (mm), resonance (Hz), downweight (g) for each 88th key, and return weight (g) for each key 182.

[0033] The information included in the performance characteristics Xa is not limited to the above examples. For example, the performance characteristics Xa may include pedal settings (differences in damper, sostenuto, muffler, shift, softness, etc.), location information of the installation location of the keyboard instrument 10, the type of building in which the installation location is located (recital hall, concert hall, salon, cathedral, club, plate, soundproof room, setting value), and reverberation measurement results (dB-Time) of the installation location.

[0034] The tuning information Xb is information relating to the tuning of the keyboard instrument 10. The tuning information Xb includes, for example, a reference pitch or temperament. The reference pitch is the pitch that serves as the reference for tuning (e.g., 440 Hz, 441 Hz, 442 Hz, etc.). The temperament is set to, for example, equal temperament, just intonation, Pythagorean temperament, or middle temperament. The tuning information Xb is input by a tuner when tuning the keyboard instrument 10. Note that the initial values ​​of each piece of information, including the tuning information Xb, are set, for example, before the keyboard instrument 10 is shipped.

[0035] The information included in the tuning information Xb is not limited to the above examples. For example, the tuning information Xb also includes information such as rapid-fire performance, key depth, aftertouch, velocity range, dynamic range, resonance, downweight, upweight, and comments. While some of the tuning information Xb overlaps with the operating characteristics Xa, the results of changes made during tuning or over time are saved. Even for acoustic pianos without a detection device 185, tuning information Xb is input and saved during tuning and used for comparison with other instruments. During tuning, the terminal device 30 is placed in a specific position, such as a music stand, and the volume or frequency measured by the terminal device 30 can be saved as tuning information Xb. The measured values ​​can also be converted using a fixed ratio and saved as tuning information Xb.

[0036] The historical information Xc is information related to the performance by the performer Ua, and is temporarily stored in the keyboard instrument 10 until it is transferred to the terminal device 30. In other words, after being transferred to the terminal device 30, the historical information Xc is deleted from the storage device 12. The historical information Xc is also expressed as information (performance log) that represents the history of the performance by the performer Ua.

[0037] Specifically, the history information Xc includes the identification information of the performer Ua, the sound source used, the mode used, the performance date and time (start time and end time), the performance location, the performance scene, the identification information of the keyboard instrument 10, the model, the training text used by the performer Ua, and the sheet music information of the music piece performed by the performer Ua.

[0038] The communication device 13 communicates with the terminal device 30. The communication device 13 communicates with the terminal device 30 via wireless communication such as Wi-Fi (registered trademark) or Bluetooth (registered trademark). For example, the communication device 13 transmits the operation characteristic Xa, tuning information Xb, and history information Xc stored in the storage device 12 to the terminal device 30. Note that the communication between the communication device 13 and the terminal device 30 may be wired communication. The communication device 13, which is separate from the keyboard instrument 10, may be connected to the keyboard instrument 10 via wire or wirelessly.

[0039] 1 records a performance by a performer Ua. Specifically, the recording system 20 includes multiple image capture devices 21 and one sound collection device 22. Note that some or all of the multiple image capture devices 21 may be mounted on the terminal device 30 or the keyboard instrument 10. Similarly, the sound collection device 22 may be mounted on the terminal device 30 or the keyboard instrument 10. The sound collection device 22 may also be mounted on any of the multiple image capture devices 21.

[0040] Each imaging device 21 is a video camera that generates a video of the performer Ua and the keyboard instrument 10 (hereinafter referred to as the "recorded video Ya"). That is, the recording system 20 generates multiple recorded videos Ya by capturing images of the performance by the performer Ua. Each imaging device 21 includes, for example, an optical system such as a photographing lens, an imaging element that receives incident light from the optical system, and a processing circuit that generates data for the recorded video Ya according to the amount of light received by the imaging element. Note that in addition to video equipment dedicated to imaging, a general-purpose information device (e.g., a smartphone or tablet terminal) equipped with an imaging function may also be used as the imaging device 21. Furthermore, the format of the data representing the recorded video Ya is arbitrary.

[0041] The multiple imaging devices 21 are installed at different positions and angles relative to the performer Ua and the keyboard instrument 10. Furthermore, the angle of view of each imaging device 21 is different. That is, each imaging device 21 captures a different part of the performer Ua or the keyboard instrument 10. The multiple imaging devices 21 generate recorded videos Ya in parallel with each other in parallel with the performance by the performer Ua. That is, the recording system 20 generates multiple recorded videos Ya that are recorded in parallel at different positions or angles.

[0042] 4 to 7 are schematic diagrams of the recorded videos Ya. The recorded videos Ya generated by the recording system 20 include a keyboard video Ya1, a performance video Ya2, a performance video Ya3, and a pedal video Ya4.

[0043] 4, the keyboard video Ya1 is a video of the keyboard 181 of the keyboard instrument 10 and the right hand HR and left hand HL of the performer Ua captured from above the keyboard instrument 10. In other words, the keyboard video Ya1 includes the keyboard 181 used to play the musical piece and both hands of the performer Ua. The keyboard video Ya1 does not include the face of the performer Ua.

[0044] 5, the performance video Ya2 is a video captured from the side of the performer Ua. The performance video Ya2 includes the performer Ua's face (e.g., a profile), the performer Ua's right hand HR and left hand HL, and the side of the keyboard instrument 10. In other words, the performance video Ya2 shows the performer Ua's facial expressions while performing the musical piece.

[0045] 6, the performance video Ya3 is a video captured from the right rear of the performer Ua. The performance video Ya3 includes the back and back of the head of the performer Ua, the right hand HR and both feet of the performer Ua, and the front of the keyboard instrument 10. The performance video Ya3 does not include the face of the performer Ua.

[0046] 7, the pedal video Ya4 is a video capturing images of the pedal 184 of the keyboard instrument 10 and the feet of the performer Ua. In other words, the pedal video Ya4 includes both feet of the performer Ua and the pedal 184 of the keyboard instrument 10. The pedal video Ya4 does not include the face of the performer Ua.

[0047] The sound collection device 22 in Fig. 1 is one or more microphones that collect sounds around the performer Ua (hereinafter referred to as "recorded sounds Yb"). Specifically, the sound collection device 22 is installed facing the keyboard instrument 10 and collects the performance sounds emitted by the keyboard instrument 10 as a result of the performance of the performer Ua as the collected sounds Yb. In addition to devices dedicated to collecting sounds, a general-purpose information device equipped with a sound collection function (e.g., a smartphone or tablet terminal) may also be used as the sound collection device 22. The format of the data representing the recorded sounds Yb is arbitrary.

[0048] Each imaging device 21 is connected to the terminal device 30 by wire or wirelessly. The recorded video Ya recorded by each imaging device 21 is transmitted to the terminal device 30. Similarly, the sound collection device 22 is connected to the terminal device 30 by wire or wirelessly. The recorded sound Yb by the sound collection device 22 is transmitted to the terminal device 30. As described above, the recording data Y representing the multiple recorded videos Ya and recorded sounds Yb is transmitted from the recording system 20 to the terminal device 30.

[0049] The terminal device 30 manages the temporal correspondence between each recorded video Ya and recorded sound Yb generated by the recording system 20 and the performance data generated by the keyboard instrument 10. Specifically, the terminal device 30 transmits synchronization data to the recording system 20 and the keyboard instrument 10. Each imaging device 21 adds synchronization data to the recorded video Ya, and each sound collection device 22 adds synchronization data to the recorded sound Yb. The keyboard instrument 10 also adds synchronization data to the performance data. Therefore, by referencing the synchronization data, the control device 31 can identify the corresponding points in time between each recorded video Ya, recorded sound Yb, and performance data. In other words, it is possible to synchronize each recorded video Ya, recorded sound Yb, and performance data D with each other. The synchronization data is, for example, LTC (Longitudinal Time Code) or MIDI time code.

[0050] 8 is a schematic diagram of a screen (hereinafter referred to as a "setting screen 60") displayed on the display device 34 for managing the connection between the recording system 20 and the terminal device 30. The setting screen 60 is displayed on the display device 34 in response to an operation by the user U on the operation device 35. After completing the settings for the recording system 20 using the setting screen 60, the performer Ua starts playing using the keyboard instrument 10.

[0051] The setting screen 60 includes a guide screen 61. The guide screen 61 is an image that guides the user U (Ua, Ub, Uc) to the position where each device of the recording system 20 should be installed. Specifically, the guide screen 61 displays a plurality of device icons 62 representing each device at the position where the device should be installed relative to the keyboard instrument 10.

[0052] The multiple device icons 62 include an imaging icon 62a and a sound collection icon 62b. The imaging icon 62a is an image representing the imaging device 21 connected to the terminal device 30. The sound collection icon 62b is an image representing the sound collection device 22 connected to the terminal device 30. By operating the operation device 35, the user U can move the imaging icon 62a to a position corresponding to the actual position of each imaging device 21 relative to the keyboard instrument 10. Similarly, the user U can move the sound collection icon 62b to a position corresponding to the actual position of the sound collection device 22 relative to the keyboard instrument 10.

[0053] While referring to the guide screen 61 , the user U sets up the recording system 20 around the keyboard instrument 10 and connects each imaging device 21 and sound collecting device 22 to the terminal device 30 .

[0054] As illustrated in Fig. 8, the setting screen 60 includes a scene selection window 63 and an equipment setting window 64. The scene selection window 63 is an image that allows the user U to select a performance scene by the performer Ua. Specifically, a plurality of performance scenes (normal performance, performance lesson, recital, street performance) with different performance purposes or situations are displayed in the scene selection window 63 as selection candidates for the user U. The user U can select one of the plurality of performance scenes by operating the operation device 35. The performance scene selected by the user U is shared between the terminal device 30 and the keyboard instrument 10.

[0055] The location where each device in the recording system 20 should be installed varies for each performance scene. Therefore, the guide screen 61 changes depending on the performance scene selected by the user U from the scene selection window 63. That is, the guide screen 61 guides the user U to the appropriate location of each device in each performance scene. Figure 8 shows the setting screen 60 when "performance training" is selected as the performance scene.

[0056] The measurement system 40 in FIG. 1 measures biometric information Z related to the player Ua. Specifically, the measurement system 40 includes a biometric sensor 41 and a center-of-gravity sensor 42. The biometric sensor 41 is a portable sensor worn on the body of the player Ua. The biometric sensor 41 is, for example, a heart rate sensor that detects the heart rate of the player Ua. The center-of-gravity sensor 42 is a sensor installed on the chair on which the player Ua sits while playing the keyboard instrument 10, and detects movement of the center of gravity of the seat. The biometric information Z of the player Ua is data including, for example, the player Ua's playing posture, level of tension (strain), heart rate, and concentration.

[0057] The playing posture is an index relating to the posture of the player Ua playing the keyboard instrument 10. The playing posture is determined by analyzing the detection results of the center of gravity sensor 42. Note that the playing posture may also be determined by analyzing a recorded video Ya of the player Ua captured by the imaging device 21.

[0058] The heart rate is detected, for example, by a heart rate sensor. The tension level is an index of the player Ua's level of tension (strain) and is calculated, for example, based on the player Ua's heart rate and the stability of his / her playing posture. The concentration level is an index of the player Ua's level of concentration on playing and is calculated, for example, based on the player Ua's heart rate and the stability of his / her playing posture. Note that the concentration level may be set to a higher value the more times a key is pressed within a specified period of time. Note that the configuration and type of the measurement system 40 are not limited to the above examples.

[0059] In the configuration described above, the terminal device 30 acquires various information from the peripheral devices (the keyboard instrument 10, the recording system 20, and the measurement system 40). For example, the control device 31 transmits an information request to the peripheral devices from the communication device 33, and receives information transmitted from the peripheral devices in response to the information request via the communication device 33. Specifically, the terminal device 30 acquires the motion characteristics Xa, tuning information Xb, and history information Xc from the keyboard instrument 10, acquires recording data Y representing a plurality of recorded videos Ya and recorded sounds Yb from the recording system, and acquires biometric information Z of the performer Ua from the measurement system 40.

[0060] As illustrated in Fig. 9, performance data D is stored in the storage device 32 of the terminal device 30. The performance data D is comprehensive data related to the performance by the performer Ua. Specifically, the performance data D includes music data corresponding to the results of detection by the detection device 185, motion characteristics Xa, tuning information Xb, history information Xc, recorded data Y, and biometric information Z. The performance data D is stored in the storage device 32 for each performance by the performer Ua (each time the performer logs in to the keyboard instrument 10). The terminal device 30 executes various processes using the performance data D.

[0061] 1 is a computer system (cloud storage) that records various types of information. Some or all of the information stored in the storage device 32 of each terminal device 30 or the storage device 12 of the keyboard instrument 10 may be stored in the recording system 90. In other words, some or all of the storage device 32 or the storage device 12 may be replaced by the recording system 90. For example, some or all of the performance data D may be transferred from a peripheral device (the keyboard instrument 10, the recording system 20, the measurement system 40) to the recording system 90, and then transmitted from the recording system 90 to the terminal device 30 in response to a request from the terminal device 30.

[0062] 10 is a block diagram illustrating an example of the functional configuration of the terminal device 30. The control device 31 executes a program stored in the storage device 32 to realize multiple functions (a characteristic acquisition unit 51, a performance analysis unit 52, and a display control unit 53) for analyzing the performance by the performer Ua.

[0063] The characteristic acquisition unit 51 acquires information about the keyboard instrument 10 (motion characteristic Xa, tuning information Xb, and history information Xc). Specifically, the characteristic acquisition unit 51 receives the motion characteristic Xa, tuning information Xb, and history information Xc transmitted from the keyboard instrument 10 via the communication device 33. As described above, the history information Xc includes music data representing the performance by the performer Ua. In other words, the characteristic acquisition unit 51 can also be expressed as an element that acquires music data. The characteristic acquisition unit 51 also acquires, from the storage device 32, a picked-up signal that records the performance of the keyboard instrument 10 by the performer Ua.

[0064] The performance analysis unit 52 identifies performance characteristics F by analyzing the performance of the keyboard instrument 10 by the performer Ua. The performance characteristics F are characteristics specific to the performance of the keyboard instrument 10 by the performer Ua. The performance characteristics F are characteristics measured from the performance of the performer Ua and include multiple pieces of information corresponding to the above-mentioned motion characteristics Xa. Specifically, the performance characteristics F include a dynamic range, a velocity range, a frequency range, a repeated key characteristic, and a key depression sensitivity.

[0065] The dynamic range is the range of volume (the ratio between maximum and minimum volume) of the sounds produced by the keyboard instrument 10 when played by the performer Ua. The performance analysis unit 52 identifies the dynamic range by comparing the depth of the key 182 with the key pressing speed (see FIG. 27). If the key pressing speed is slow relative to the reach to the deepest part of the key 182, it is determined to be pianissimo, and if it is fast, it is determined to be fortissimo. The speed (volume) at the time of determination can be obtained as dynamic range data.

[0066] In addition, by recording the key pressing speed for each note when playing the same song for each device and location, it is possible to compare performances on different devices and in different locations.It is also possible to compare performances by different performers Ua, such as performances of the same song by a student and a teacher.

[0067] The performance analysis unit 52 can also determine musical expressions associated with the performance, such as staccato, legato, trills, and ornaments. For example, in a staccato performance, which is a series of short chords, the performance analysis unit 52 compares the pitch of the performance before and after the staccato to determine whether a beat occurs between the chords. In a legato performance, the performance characteristics of the performer can be determined by analyzing the performance data D, such as by comparing the performance before and after the staccato to determine whether the previous note overlaps the next note. The determination of performance expressions is also disclosed, for example, in Japanese Patent Application Laid-Open No. 2006-106313.

[0068] Sensors provided on the keyboard 181 can measure the key depression speed, which indicates the speed at which the key 182 is depressed downward, and can also measure the key release speed, which indicates the speed at which the depressed key 182 returns upward. In performance, not only the force with which the key 182 is pressed but also the action of releasing the finger from the key 182 are important performance characteristics F. If the balance between key depression and key release is poor, the performance pitch and sound output time will be disrupted, and by detecting this disruption, the quality of the performance can be determined using the key depression speed, key release speed, and key depth.

[0069] The velocity range is the range of key pressing speed (key pressing strength) by the performer Ua. The performance analysis unit 52 identifies the velocity range by analyzing the performance data D. Specifically, the velocity range is identified as the range between the maximum and minimum velocity values ​​specified by the performance data D.

[0070] The frequency range is the range of frequencies of the performance sounds that are generated by the keyboard instrument 10 when played by the performer Ua. The performance analysis unit 52 identifies the frequency range by analyzing the collected sound signal supplied from the sound collection device 37.

[0071] The repeated keying characteristic is the maximum number of times a key is pressed per unit time by the performer Ua. The performance analysis unit 52 identifies the repeated keying characteristic by analyzing the performance data D. The key pressing sensitivity is identified by analyzing the performance data D and the sound pickup signal supplied from the sound pickup device 37. The performance analysis unit 52 may also identify the key pressing sensitivity by analyzing a video captured of the performer Ua playing the keyboard instrument 10.

[0072] Half-touch can also be detected when determining the characteristics of repeated strokes. Half-touch is a technique that is possible on grand pianos and some electronic instruments, and is a function that allows the next key to be pressed before the key has fully risen. If the same note can be produced without the key having fully returned while measuring the key depth, half-touch performance is being performed. It is possible to distinguish and record whether half-touch performance has been performed along with the number of repeated strokes.

[0073] Aftertouch can also be detected while playing ornaments, trills, and pianissimos. Aftertouch is a technique available on grand pianos and some electronic instruments. When a key is pressed, a sound is produced 2-3 mm before the key's depth. It is a performance expression related to touch manipulation around this sound point. Pressing the key harder than necessary after sound production not only generates lower-level noise but also stops the body vibration. Aftertouch is recognized as a performance technique when the key depth is measured and the sound is produced at the appropriate sound point. Along with determining whether ornaments, trills, or pianissimos are being performed, aftertouch performance can be recorded separately.

[0074] 10 displays the movement characteristic Xa and the performance characteristic F on the display device 34. For example, the display control unit 53 displays the movement characteristic Xa and the performance characteristic F on the display device 34 in a manner that allows the user U to compare them. Specifically, the display control unit 53 displays information on the correspondence between the movement characteristic Xa and the performance characteristic F on the display device 34.

[0075] 11 and 12 are schematic diagrams of reference screens G (G1, G2) displayed on the display device 34 for comparing the motion characteristic Xa with the performance characteristic F. The reference screen G1 of Fig. 11 and the reference screen G2 of Fig. 12 are displayed, for example, in parallel. Note that the reference screen G1 and the reference screen G2 may be selectively displayed in response to an operation by the user U on the operation device 35, for example.

[0076] 11 is a schematic diagram of a reference screen G1 for comparing the dynamic range and velocity between the movement characteristic Xa and the performance characteristic F. The reference screen G1 includes a comparison image E and a comment C. The comparison image E is an image that displays the dynamic range and velocity in parallel for each of the movement characteristic Xa and the performance characteristic F. The dynamic range of the performance characteristic F may be updated in real time in parallel with the performance of the keyboard instrument 10 by the performer Ua, or may be updated at predetermined intervals in the music piece (for example, for each phrase).

[0077] 11 displays the dynamic range and velocity for the movement characteristic Xa, the performance characteristic F, and also for the reference characteristic R. The reference characteristic R is an exemplary or standard characteristic related to the performance of a piece of music, and is stored in advance in, for example, the storage device 32. The dynamic range and velocity for each of the movement characteristic Xa, the performance characteristic F, and the reference characteristic R are displayed on a common number line.

[0078] Therefore, the performer Ua can visually and intuitively grasp the relationship between the dynamic range and velocity (operation characteristic Xa) specific to the keyboard instrument 10, the dynamic range and velocity (performance characteristic F) related to his / her own performance, and the exemplary or standard dynamic range and velocity (reference characteristic R) related to the performance of a musical piece.

[0079] The comment C on the reference screen G1 is a character string relating to the relationship between the movement characteristic Xa and the performance characteristic F. For example, a plurality of comments C relating to different relationships between the movement characteristic Xa and the performance characteristic F are stored in the storage device 32. The display control unit 53 displays, on the reference screen G, the comment C corresponding to the relationship between the movement characteristic Xa and the performance characteristic F, among the plurality of comments C stored in the storage device 32. With the above configuration, the performer Ua can easily understand areas for improvement in his or her own performance.

[0080] 12 is a schematic diagram of a reference screen G2 for comparing the velocity range and frequency range between the motion characteristic Xa, the performance characteristic F, and the reference characteristic R. The reference screen G2 includes a first axis A1 and a second axis A2 that are orthogonal to each other.

[0081] The first axis A1 (vertical axis) is a number line representing the velocity range. Specifically, the velocity ranges for the motion characteristic Xa, performance characteristic F, and reference characteristic R are displayed in parallel along the first axis A1. Therefore, the performer Ua can visually and intuitively grasp the relationship between the velocity range inherent to the keyboard instrument 10, the velocity range related to his or her own performance, and the exemplary or standard velocity range.

[0082] The second axis A2 (horizontal axis) is a number line representing the frequency range. Specifically, the frequency ranges for the motion characteristic Xa, performance characteristic F, and reference characteristic R are displayed in parallel along the second axis A2. Therefore, the performer Ua can visually and intuitively grasp the relationship between the frequency range inherent to the keyboard instrument 10 and the frequency range related to his or her own performance.

[0083] As described above, in the first embodiment, the motion characteristic Xa related to the keyboard instrument 10 and the performance characteristic F related to the performance using the keyboard instrument 10 are displayed. Therefore, the performer Ua can visually and intuitively grasp the performance characteristic F of his / her own with respect to the motion characteristic Xa of the keyboard instrument 10.

[0084] 13 is a flowchart of the process executed by the control device 31 to analyze the performance of the performer Ua (hereinafter referred to as the "performance analysis process"). For example, the performance management process is executed in response to an operation of the operation device 35 by the performer Ua.

[0085] When the performance management process is started, the control device 31 (characteristics acquisition unit 51) acquires the motion characteristics Xa, tuning information Xb, and history information Xc from the keyboard instrument 10 (Sa1). Specifically, the characteristics acquisition unit 51 receives the motion characteristics Xa, tuning information Xb, and history information Xc transmitted from the keyboard instrument 10 via the communication device 33.

[0086] The control device 31 (display control unit 53) displays a performance history screen H on the display device 34 (Sa2). As illustrated in Fig. 14, the performance history screen H includes information such as the performance location and the equipment used for each date on which the performer Ua played the keyboard instrument 10. The control device 31 displays the performance history screen H on the display device 34 using the history information Xc acquired from the keyboard instrument 10.

[0087] The performer Ua can operate the operation device 35 to select one of a plurality of dates (hereinafter referred to as the "selected date") on the performance history screen H. When an instruction for the selected date is received from the performer Ua, the control device 31 (characteristics acquisition unit 51) acquires the picked-up signal stored in the storage device 32 for the selected date (Sa3).

[0088] The control device 31 (performance analysis unit 52) ​​analyzes the performance of the performer Ua on the selected day to identify the performance characteristic F (Sa4). Specifically, the control device 31 analyzes the history information Xc for the selected day and the sound pickup signal for the selected day to identify the performance characteristic F. The performance characteristic F identified by the analysis is stored in the storage device 32 and can be retrieved from the storage device 32 as needed.

[0089] The control device 31 (display control unit 53) generates image data for a reference screen G (G1, G2) representing the movement characteristic Xa and the performance characteristic F (Sa5). The control device 31 (display control unit 53) outputs the image data to the display device 34, thereby displaying the reference screen G on the display device 34 (Sa6).

[0090] B: Second Embodiment A second embodiment will be described. Note that, for elements in the following exemplary aspects that have the same functions as those in the first embodiment, the same reference numerals as those in the first embodiment will be used, and detailed descriptions of each element will be omitted as appropriate.

[0091] Fig. 15 is a schematic diagram of a reference screen G3 displayed on the display device 34 in the second embodiment. The reference screen G3 in Fig. 15 is displayed together with the reference screens G (G1, G2) exemplified in the first embodiment. Note that each reference screen G (G1, G2, G3) may be selectively displayed on the display device 34. In the second embodiment, the display of the reference screen G1 and the reference screen G2 may be omitted.

[0092] The reference screen G3 includes the performance characteristics F for each specific period (hereinafter referred to as the "unit period") and the motion characteristics Xa of the keyboard instrument 10. The unit period is the period of time during which the performer Ua plays the keyboard instrument 10 that is the subject of display. For example, one day corresponding to a specific day of the week is exemplified as the unit period. Note that FIG. 15 illustrates an example of the reference screen G3 that displays the dynamic range of the performance characteristics F for each unit period. Note that the performance characteristics F for each unit period (each day of the week) are displayed based on, for example, the maximum and minimum values ​​of the dynamic range of the performance within that day.

[0093] The performance analysis unit 52 identifies a performance characteristic F for each of a plurality of unit periods. Specifically, the performance analysis unit 52 identifies the performance characteristic F for each unit period by analyzing history information Xc (e.g., music data) for the unit period and the picked-up signal for that unit period. The display control unit 53 displays, on the display device 34, a reference screen G3 including the operation characteristic Xa of the keyboard instrument 10 and the performance characteristic F for each unit period identified by the performance analysis unit 52, as shown in the example of FIG. 15 .

[0094] The second embodiment also achieves the same effects as the first embodiment. Furthermore, in the second embodiment, the performance characteristic F of the performer Ua is displayed for each of a plurality of unit periods, allowing the performer Ua to visually confirm the temporal fluctuations of the performance characteristic F. For example, the performer Ua can visually grasp how the performance characteristic F changes over time as the performer Ua improves in their performance.

[0095] In the above explanation, the performance characteristics F relating to the overall performance by the performer Ua are displayed together with the movement characteristics Xa, but for example, the performance characteristics F relating to the performance of a specific piece of music may be displayed together with the movement characteristics Xa on the display device 34. For example, the performance characteristics F relating to the performance of a set piece instructed to the performer Ua in a music class may be displayed on the display device 34.

[0096] Furthermore, the motion characteristic Xa and the performance characteristic F may be displayed on the display device 34 for each of a plurality of different types of keyboard instruments 10. For example, in the reference screen G3 of Fig. 15, the performance characteristic F for Friday may be the performance characteristic F when the performer Ua plays a grand piano in a music studio, and the performance characteristic F for days other than Friday may be the performance characteristic F when the performer Ua plays an upright piano at home.

[0097] 15 shows an example of a reference screen G3 including a comment C regarding the relationship between the movement characteristic Xa and the performance characteristic F. Specifically, a plurality of comments C regarding different relationships between the movement characteristic Xa and the performance characteristic F are stored in the storage device 32, and among the plurality of comments C, a comment C corresponding to the performance by the performer Ua is displayed. FIG. 15 shows an example of a comment C praising the performer for being able to play with a wide dynamic range, while also urging attention to the key pressing speed (strength).

[0098] C: Third Embodiment Fig. 16 is a block diagram illustrating the configuration of an information processing system 100 according to the third embodiment. The information processing system 100 of the third embodiment includes a terminal device 30 and multiple keyboard instruments 10 (10A, 10B, 10C). Each keyboard instrument 10 is a performance device installed in a different location. For example, the keyboard instrument 10A is installed in the home of a performer Ua, and the keyboard instrument 10B is installed in a music studio to which the performer Ua belongs.

[0099] The total number of keyboard instruments 10 is not limited to two. For example, in addition to the keyboard instruments 10A and 10B shown in Fig. 16, a keyboard instrument 10C installed at the venue of a concert in which performer Ua will participate may also be included in the information processing system 100. The keyboard instrument 10A is an example of a "first performance device," and the keyboard instrument 10B is an example of a "second performance device."

[0100] The keyboard instrument 10A is an upright piano with a mute function installed at home, and when mute is performed, the hammer action is not performed, but rather a device with the function of a so-called electronic musical instrument that detects the movement of the keys with a sensor, processes the detected signal with a processor, and outputs sound. The keyboard instrument 10B is a grand piano with hammer action, and has sensors on the keys so that performance data D can be acquired along with the output of the performance sound of an acoustic piano by hammer action. The keyboard instrument 10C installed at the venue of the recital is a grand piano with hammer action, but has no sensors on the keys, and acquires performance data D by a sound pickup device 37 of the terminal device 30.

[0101] The storage device 12 of each keyboard instrument 10 stores the motion characteristic Xa, tuning information Xb, and history information Xc of that keyboard instrument 10. Specifically, the storage device 12 of the keyboard instrument 10A stores the motion characteristic Xa_A, tuning information Xb_A, and history information Xc_A. The storage device 12 of the keyboard instrument 10B stores the motion characteristic Xa_B, tuning information Xb_B, and history information Xc_B. The motion characteristic Xa_A of the keyboard instrument 10A is different from the motion characteristic Xa_B of the keyboard instrument 10B. Similarly, the tuning information Xb_A of the keyboard instrument 10A is different from the tuning information Xb_B of the keyboard instrument 10B. The motion characteristic Xa_A is an example of a "first motion characteristic," and the motion characteristic Xa_B is an example of a "second motion characteristic."

[0102] The performer Ua can carry the terminal device 30 to the location of the keyboard instrument 10A (e.g., at home) and play the keyboard instrument 10A, and can also carry the terminal device 30 to the location of the keyboard instrument 10B (e.g., a music studio) and play the keyboard instrument 10B. The performance conditions, such as the performance date and time, performance location, and music piece performed by the performer Ua, differ between the keyboard instruments 10A and 10B. Therefore, the history information Xc_A of the keyboard instrument 10A and the history information Xc_B of the keyboard instrument 10B differ.

[0103] Furthermore, when the performer Ua plays the keyboard instrument 10A, a picked-up signal representing the performance sound of the keyboard instrument 10A is stored in the storage device 32 of the terminal device 30. Similarly, when the performer Ua plays the keyboard instrument 10B, a picked-up signal representing the performance sound of the keyboard instrument 10B is stored in the storage device 32 of the terminal device 30.

[0104] 17 is a schematic diagram of a reference screen G4 displayed on the display device 34 in the third embodiment. The reference screen G4 in FIG. 17 is displayed together with the reference screens G (G1, G2) exemplified in the first or second embodiment. Note that each reference screen G (G1 to G4) may be selectively displayed on the display device 34. In the third embodiment, the display of screens (G1 to G3) other than the reference screen G4 may be omitted.

[0105] The reference screen G4 is an image including a comparison image E (EA, EB) for each keyboard instrument 10 (10A, 10B) and a comment C regarding the performance. The comparison image EA contrasts the motion characteristic Xa_A of the keyboard instrument 10A with the performance characteristic F_A when the performer Ua plays the keyboard instrument 10A. The comparison image EB contrasts the motion characteristic Xa_B of the keyboard instrument 10B with the performance characteristic F_B when the performer Ua plays the keyboard instrument 10B. Because the motion characteristic Xa and the installation environment differ for each keyboard instrument 10, the performance characteristic F_A related to the performance of the keyboard instrument 10A differs from the performance characteristic F_B related to the performance of the keyboard instrument 10B. Note that the performance characteristic F_A is an example of a "first performance characteristic," and the performance characteristic F_B is an example of a "second performance characteristic."

[0106] The characteristic acquisition unit 51 of the third embodiment acquires the motion characteristic Xa_A, tuning information Xb_A, and history information Xc_A from the keyboard instrument 10A, and acquires the motion characteristic Xa_B, tuning information Xb_B, and history information Xc_B from the keyboard instrument 10B. The characteristic acquisition unit 51 also acquires, from the storage device 32, a picked-up sound signal recorded when the performer Ua plays the keyboard instrument 10A, and a picked-up sound signal recorded when the performer Ua plays the keyboard instrument 10B.

[0107] The performance analysis unit 52 of the third embodiment determines the performance characteristic F_A by analyzing the performance results (history information Xc_A and picked-up sound signals) of the performer Ua playing the keyboard instrument 10A. Similarly, the performance analysis unit 52 determines the performance characteristic F_B by analyzing the performance results (history information Xc_B and picked-up sound signals) of the performer Ua playing the keyboard instrument 10B.

[0108] As described above, the keyboard instruments 10A and 10B are installed in different locations. Therefore, the performance characteristics F_A and F_B can also be expressed as characteristics related to the performance of the performer Ua at different locations. Specifically, the performance characteristic F_A is the performance characteristic F of the performer Ua at a first location (e.g., at home), and the performance characteristic F_B is the performance characteristic F of the performer Ua at a second location (e.g., a music studio) different from the first location.

[0109] The display control unit 53 displays a reference screen G4 including a comparison image EA representing the motion characteristic Xa_A and performance characteristic F_A of the keyboard instrument 10A and a comparison image EB representing the motion characteristic Xa_B and performance characteristic F_B of the keyboard instrument 10B on the display device 34. That is, the display control unit 53 of the third embodiment displays the relationship between the motion characteristic Xa_A and performance characteristic F_A and the relationship between the motion characteristic Xa_B and performance characteristic F_B.

[0110] The third embodiment also achieves the same effects as the first embodiment. Furthermore, in the third embodiment, the performer Ua can visually compare the motion characteristic Xa_A of the keyboard instrument 10A with the motion characteristic Xa_B of the keyboard instrument 10B, as well as visually compare the performance characteristic F_A when playing the keyboard instrument 10A with the performance characteristic F_B when playing the keyboard instrument 10B. Therefore, the performer Ua can easily grasp the points to be mindful of when playing each of the keyboard instruments 10A and 10B.

[0111] The comments C on the reference screen G4 are character strings relating to the relationship between the motion characteristic Xa and the performance characteristic F for each keyboard instrument 10 and the relationship between the performance characteristics F between different keyboard instruments 10. Specifically, a comment C is stored in the storage device 32 for each combination of the relationship between the motion characteristic Xa_A and the performance characteristic F_A, the relationship between the motion characteristic Xa_B and the performance characteristic F_B, and the relationship between the performance characteristic F_A and the performance characteristic F_B. The display control unit 53 displays, on the reference screen G4, the comments C corresponding to the relationship between the motion characteristic Xa and the performance characteristic F for each keyboard instrument 10, and the relationship between the performance characteristic F_A and the performance characteristic F_B, from among the multiple comments C stored in the storage device 32. This allows the performer Ua to easily grasp the difference in the performance characteristic F due to differences in performance location. For example, the performer Ua can grasp the difference between the performance characteristic F_A when playing the keyboard instrument 10A at home and the performance characteristic F_A when playing the keyboard instrument 10A in a music studio.

[0112] D: Fourth Embodiment An information processing system 100 in the fourth embodiment includes a terminal device 30 and a plurality of keyboard instruments 10 (10A, 10B), similar to the third embodiment illustrated in Fig. 16. A performer Ua can selectively play the keyboard instrument 10A or the keyboard instrument 10B.

[0113] As in the third embodiment, the characteristic acquisition unit 51 of the fourth embodiment acquires the operating characteristic Xa_A, tuning information Xb_A, and history information Xc_A from the keyboard instrument 10A, and also acquires the operating characteristic Xa_B, tuning information Xb_B, and history information Xc_B from the keyboard instrument 10B.

[0114] Similarly to the third embodiment, the performance analysis unit 52 of the fourth embodiment determines the performance characteristic F_A by analyzing the performance results (history information Xc_A and picked-up sound signals) of the performer Ua playing the keyboard instrument 10A. Similarly, the performance analysis unit 52 determines the performance characteristic F_B by analyzing the performance results (history information Xc_B and picked-up sound signals) of the performer Ua playing the keyboard instrument 10B. As described above in relation to the third embodiment, the performance characteristic F_A and the performance characteristic F_B are performance characteristics F related to the performance of different keyboard instruments 10. The performance characteristic F_A and the performance characteristic F_B can also be expressed as characteristics related to the performance of the performer Ua in different locations.

[0115] The display control unit 53 of the fourth embodiment displays the performance characteristics F_A and F_B on the display device 34. Specifically, the display control unit 53 displays a reference screen G5 on the display device 34 for comparing the performance characteristics F_A and F_B.

[0116] Fig. 18 is a schematic diagram of reference screen G5. For example, reference screen G5 in Fig. 18 is displayed together with the reference screens G (G1 to G4) exemplified in the first to third embodiments. Note that each reference screen G (G1 to G5) may be selectively displayed on the display device 34. In the fourth embodiment, display of screens (G1 to G4) other than reference screen G5 may be omitted.

[0117] Each performance characteristic F (Q_A, Q_B) includes evaluation values ​​for each of a number of evaluation items. Specifically, the performance characteristics F include seven evaluation items: repeated keystroke characteristics, strong keystrokes (fortissimo), soft keystrokes (pianissimo), aftertouch, dynamic range, key-on speed, and key-off speed.

[0118] The reference screen G5 includes an evaluation image G51 and comments C. The evaluation image G51 is a graph showing evaluation values ​​for each of a plurality of evaluation items. Specifically, the evaluation image G51 is a polygonal radar chart in which evaluation values ​​for each evaluation item are expressed by a plurality of number lines extending radially. The performance characteristic F_A and the performance characteristic F_B are displayed overlapping each other. That is, the display control unit 53 displays the evaluation values ​​for each of the performance characteristics F_A and F_B for each evaluation item. As described above, the reference screen G5 of the fourth embodiment differs from the reference screens G of the first to third embodiments in that it does not include a display related to the motion characteristic Xa of the keyboard instrument 10.

[0119] As described above, in the fourth embodiment, a performance characteristic F_A related to the performance of the keyboard instrument 10A and a performance characteristic F_B related to the performance of the keyboard instrument 10B are displayed. Therefore, the performer Ua can visually and intuitively grasp the change in the performance characteristic F caused by the difference between the keyboard instruments 10. In particular, in the fourth embodiment, the evaluation value of the performance characteristic F_A and the evaluation value of the performance characteristic F_B are displayed for each of a plurality of evaluation items related to performance. Therefore, the performer Ua can grasp the change in the performance characteristic F caused by the difference between the keyboard instruments 10 from a plurality of perspectives (evaluation items).

[0120] The comment C on the reference screen G5 is a character string related to the relationship between the performance characteristic F_A and the performance characteristic F_B. For example, a comment C is stored in the storage device 32 for each relationship (e.g., large or small) between the evaluation value of each evaluation item of the performance characteristic F_A and the evaluation value of each evaluation item of the performance characteristic F_B. The display control unit 53 displays, on the reference screen G5, the comment C corresponding to the relationship between the evaluation value of each evaluation item of the performance characteristic F_A and the evaluation value of each evaluation item of the performance characteristic F_B, among the multiple comments C stored in the storage device 32. This allows the performer Ua to easily grasp the difference between the performance characteristic F_A and the performance characteristic F_B. The sound mode included in the comment C in FIG. 18 is, for example, an operating mode in which the sound generation mechanism 183 is set to a mute mode and the performance sound generated by the sound source device 16 is radiated from the soundboard of the keyboard instrument 10.

[0121] In the fourth embodiment, the upright piano at home has inferior repetition characteristics compared to the grand piano at the music studio, but the hammer action of the upright piano is inferior to the hammer action of the grand piano in terms of repetition performance. Comment C conveys that a comparison of the equipment used and performance characteristics shows that this is not a problem of the performer's skill. In this way, by distinguishing between the location and the equipment, and acquiring and comparing performance data D, it is possible to select the appropriate equipment, encourage appropriate practice, and prevent the performer Ua from continuing to play at an unreasonable level and causing damage.

[0122] E: Fifth Embodiment Fig. 19 is a schematic diagram of a reference screen G6 displayed on the display device 34 in the fifth embodiment. The reference screen G6 in Fig. 19 is displayed together with the reference screens G (G1 to G5) exemplified in the first to fourth embodiments. Note that each reference screen G (G1 to G6) may be selectively displayed on the display device 34. In the fifth embodiment, the display of screens (G1 to G5) other than the reference screen G6 may be omitted.

[0123] The reference screen G6 includes an evaluation image G61 and a comment C. Similar to the evaluation image G51 of the fifth embodiment, the evaluation image G61 is a graph showing evaluation values ​​for each of a plurality of evaluation items. The performance characteristic F_A and the performance characteristic F_B are displayed overlapping each other. That is, the display control unit 53 displays the evaluation values ​​for each of the performance characteristics F_A and F_B for each evaluation item. Therefore, the fifth embodiment also achieves the same effects as the fourth embodiment.

[0124] The playing posture is an index of whether the posture of the performer Ua playing the keyboard instrument 10 is appropriate. The playing posture is determined, for example, by analyzing a recorded video Ya of the performer Ua's performance. The playing posture may also be evaluated based on the output result of the center of gravity sensor 42 that constitutes the measurement system 40.

[0125] The heart rate is detected, for example, by a biosensor 41 worn by the player Ua. The tension level is an index of the player Ua's level of tension (strain) and is calculated, for example, based on the player Ua's heart rate and the stability of his / her playing posture. The concentration level is an index of the player Ua's level of concentration on playing and is calculated, for example, based on the player Ua's heart rate and the stability of his / her playing posture. Note that the concentration level may be set to a larger value the more times a key is pressed within a specified period of time.

[0126] 18 and 19 can be specified by the user U. The item specification screen allows selection and setting for each item, such as performance characteristics, performance expression, and biometric information. In addition, it is possible to specify a piece of music to be played, collect data for the period during which a specific piece of music is being played, and display the evaluation results, or to convey the items that the user U wants to know by analyzing, sorting, and displaying information collected by the terminal device 30, such as location and device selection.

[0127] As in the fourth embodiment ( FIG. 18 ), the comment C on the reference screen G6 is a character string relating to the relationship between the performance characteristic F_A and the performance characteristic F_B. For example, a comment C is stored in the storage device 32 for each relationship (e.g., large or small) between the evaluation value of each evaluation item of the performance characteristic F_A and the evaluation value of each evaluation item of the performance characteristic F_B. Of the multiple comments C stored in the storage device 32, the display control unit 53 displays on the reference screen G6 a comment C that corresponds to the relationship between the evaluation value of each evaluation item of the performance characteristic F_A and the evaluation value of each evaluation item of the performance characteristic F_B.

[0128] F: Sixth Embodiment Fig. 20 is a schematic diagram of a reference screen G7 displayed on the display device 34 in the sixth embodiment. The reference screen G7 in Fig. 20 is displayed together with the reference screens G (G1 to G6) exemplified in the first to fifth embodiments. Note that each reference screen G (G1 to G7) may be selectively displayed on the display device 34. In the sixth embodiment, the display of screens (G1 to G6) other than the reference screen G7 may be omitted.

[0129] The reference screen G7 is an image showing a history of mistakes (hereinafter referred to as "performance mistakes") made by the performer Ua while playing a piece of music. The reference screen G7 includes a history image G71 for each of the multiple keyboard instruments 10 played by the performer Ua. In other words, a history image G71 is displayed for each location where a keyboard instrument 10 is installed.

[0130] The history image G71 of each keyboard instrument 10 is an image representing a portion where a performance error occurred when the performer Ua played a piece of music using that keyboard instrument 10. Specifically, the history image G71 is composed of a time series of a plurality of unit images G72 corresponding to different sections of the music. The section represented by one unit image G72 is, for example, one measure of the music.

[0131] Of the multiple unit images G72 constituting the history image G71, the unit images G72 corresponding to sections in which performance errors occurred are displayed in a manner different from the unit images G72 in sections in which no performance errors occurred. The display manner of the unit images G72 refers to visually distinguishable image characteristics. For example, the display color, pattern (design), size, or shape of the unit images G72 are included in the concept of "display manner." The display color is defined by hue (tone), saturation, or brightness (gradation). As explained above, of the multiple unit images G72 constituting the history image G71, the unit images G72 corresponding to sections in which performance errors by the performer Ua occurred are displayed in an emphasized manner.

[0132] As described above, the locations where performance errors occurred for each of the multiple keyboard instruments 10 are displayed on a common time axis, so the performer Ua can grasp the locations where performance errors are likely to occur when playing each keyboard instrument 10. For example, the performer Ua can grasp the tendency for performance errors to occur for each location where each keyboard instrument 10 is installed.

[0133] The reference screen G7 also includes a difficulty level image composed of a plurality of unit images G72. The difficulty level image represents a section of the musical piece that is difficult to play. Specifically, among the plurality of unit images G72 in the difficulty level image, unit images G72 corresponding to sections where the difficulty level of play exceeds a predetermined threshold are highlighted compared to the other unit images G72. Therefore, the performer Ua can visually and intuitively grasp the relationship between the difficult sections of the musical piece and sections where the performer Ua is likely to make performance mistakes.

[0134] An instruction image G73 is displayed for a section in which a performance error occurred among multiple sections of the music piece. The performer Ua can select an instruction image G73 by operating the operation device 35. When an instruction image G73 is selected, the display control unit 53 displays a comment C regarding the section indicated by the instruction image G73 on the reference screen G7.

[0135] The user U can select any one of the multiple unit images G72 on the reference screen G7 by operating the operation device 35. When a unit image G72 corresponding to "home" or "classroom" is selected from the multiple unit images G72, the display control unit 53 may display an image corresponding to the unit image G72 (for example, the recorded video Ya or the reference images G1 to G6) on the display device 34.

[0136] G: Seventh Embodiment A terminal device 30 (30a, 30b, 30c) of the seventh embodiment generates a video (hereinafter referred to as an "edited video V") representing a performance by a performer Ua, using performance data D stored in a storage device 32. Specifically, the edited video V is generated using a plurality of recorded videos Ya and recorded sounds Yb included in the recording data Y.

[0137] 21 is an explanatory diagram of edited video V. Edited video V is composed of video Va and audio Vb. Video Va and audio Vb are played back in parallel with each other. Audio Vb is recorded sound Yb (i.e., the performance sound of the keyboard instrument 10) recorded by the sound collection device 22. Audio Vb may be generated by performing various types of acoustic processing on recorded sound Yb (for example, adding effects or removing noise).

[0138] The video Va is a video in which multiple partial videos Vq (Vq1, Vq2, ...) corresponding to different phrase periods Q (Q1, Q2, ...) on the time axis are arranged in chronological order. Each phrase period Q is a section (musical phrase) that constitutes a musical unit in a song. The partial video Vq corresponding to each phrase period Q is part or all of one of the multiple recorded videos Ya (Ya1 to Ya4) recorded by the recording system 20 in that phrase period Q. As can be understood from the above explanation, the terminal device 30 generates an edited video V in which the recorded videos Ya are switched sequentially for each phrase period Q of the song. In other words, in the edited video V, the imaging position and direction change at the boundaries of the phrase periods Q of the song.

[0139] The display device 34 displays a moving image Va of the edited moving image V. The sound emitting device 36 reproduces an audio Vb of the edited moving image V. That is, the display device 34 and the sound emitting device 36 function as a reproduction device that reproduces the edited moving image V.

[0140] 22 is a block diagram illustrating an example of the functional configuration of each terminal device 30. The control device 31 executes a program stored in the storage device 32 to realize multiple functions (music analysis unit 55, video editing unit 56, and playback control unit 57) for generating the edited video V.

[0141] The music analysis unit 55 identifies the phrase period Q of the music piece by analyzing the performance data D. Specifically, the music analysis unit 55 identifies the phrase period Q by using the performance data D and reference data. The reference data is data representing the musical score of the music piece and is stored in the storage device 32. The reference data is, for example, a time series of event data conforming to the MIDI standard. Specifically, the reference data specifies the pitch (note number) and intensity (velocity) for each of the multiple notes that make up the music piece. Note that any known technology may be employed to identify the phrase period Q by the music analysis unit 55.

[0142] The video editing unit 56 generates an edited video V by selecting a plurality of recorded videos Ya in accordance with a phrase period Q. Specifically, as described above with reference to FIG. 21 , the video editing unit 56 generates a video Va in which a partial video Vq is sequentially switched to each of a plurality of recorded videos Ya for each phrase period Q of a musical piece. That is, in the video Va, the partial videos Vq are switched at the times specified by each period information T (i.e., the start and end points of each phrase period Q). That is, editing by the video editing unit 56 includes a process of selecting different recorded videos Ya from the plurality of recorded videos Ya as the partial videos Vq in one phrase period Q and the immediately following phrase period Q. For example, as illustrated in FIG. 21 , different recorded videos Ya are selected as the partial videos Vq in phrase period Q1 and phrase period Q2. The video editing unit 56 generates an edited video V by adding recorded sound Yb as audio Vb to the video Va.

[0143] 22 plays back the edited video V generated by the video editing unit 56. Specifically, the playback control unit 57 displays a video Va of the edited video V on the display device 34, and emits an audio Vb of the edited video V from the sound emission device 36.

[0144] As explained above, in the seventh embodiment, an edited video V is generated in accordance with a phrase period Q identified by analyzing the performance data D, so that an edited video V suitable for a piece of music can be generated without placing an excessive burden on the user. In particular, in the seventh embodiment, different recorded videos Ya are selected as partial videos Vq in two consecutive phrase periods Q, so that a suitable edited video V can be generated in which the partial videos Vq switch in synchronization with musical divisions in the piece of music (i.e., the start or end point of a phrase period Q).

[0145] H: Eighth Embodiment As described above, performance data D is stored in the storage device 32 for each performance by the performer Ua. That is, multiple pieces of performance data D are accumulated in chronological order in the storage device 32 as the performer Ua practices playing the keyboard instrument 10. For example, multiple pieces of performance data D representing past performances by the performer Ua, such as practice at a music school, practice at the performer Ua's home, and performances at recitals, are stored in the storage device 32. The video editing unit 56 may generate an edited video V using the multiple pieces of performance data D representing past performances by the performer Ua.

[0146] The video editing unit 56 of the eighth embodiment selects from the storage device 32 a plurality of pieces of performance data D corresponding to a performance of a specific piece of music (hereinafter referred to as a “target piece of music”) from the plurality of pieces of performance data D stored in the storage device 32. The target piece of music is, for example, a piece of music that the performer Ua will play at a recital.

[0147] The video editing unit 56 generates an edited video V by editing recorded videos Ya of multiple performance data D corresponding to the target music piece. Specifically, the video editing unit 56 generates the edited video V by selecting one of the multiple recorded videos Ya for each phrase period Q of the target music piece. For example, the multiple recorded videos Ya are selected sequentially in chronological order. Therefore, an edited video V is generated in which scenes of the performer Ua performing the target music piece are switched sequentially. Note that the conditions for switching between the recorded videos Ya are the same as those in the above-mentioned embodiments.

[0148] The eighth embodiment also achieves the same effects as the seventh embodiment. In the eighth embodiment, an edited video V is generated from a plurality of recorded videos Ya related to a target piece of music. Therefore, an edited video V suitable for recording, for example, a child's growth, can be automatically generated, showing the progress of the performer Ua's improvement in playing the target piece of music or scenes of the performer Ua's past performances.

[0149] I: Modifications Specific modifications that can be added to the above-mentioned embodiments are exemplified below. Two or more embodiments arbitrarily selected from the following examples may be combined as appropriate within the scope of not mutually contradicting each other.

[0150] (1) The reference screen G displayed on the display device 34 is not limited to the images exemplified in the above-described embodiments. For example, the display control unit 53 may display the reference screen G8 illustrated in FIG. 23 on the display device 34. The reference screen G8 includes the biometric information Z of the performer Ua in addition to the musical score G81 of the piece of music performed by the performer Ua. Specifically, the reference screen G8 includes a heartbeat waveform G82A when the performer Ua plays the piece of music on the keyboard instrument 10A and a heartbeat waveform G82B when the performer Ua plays the piece of music on the keyboard instrument 10B. Furthermore, the reference screen G8 includes an electromyogram G83 when the performer Ua plays the piece of music. With the above configuration, the physical state of the performer Ua while playing the piece of music can be visually and intuitively grasped.

[0151] The reference screen G9 illustrated in FIGS. 24 and 25 includes not only the score G91 of the piece of music performed by the performer Ua, but also motion characteristics such as key on / off speed, strength / weakness of key strokes, and repeated stroke characteristics obtained by the control device 11. Furthermore, it can also display the musical expressions to be expressed in the score, as assigned in the text, and performance expressions added by music school teachers as challenges for students. Specifically, the reference screen G9 in FIGS. 24 and 25 displays staccato, pianissimo, and key release speed as performance expressions to be challenged. During or after the performance of the piece of music based on the score G91, the motion characteristics of the performer Ua playing the piece on the keyboard instrument 10A are displayed in accordance with the notes. It also displays whether the performance expressions and challenges on the score are being expressed by the performer's performance, allowing the performer to review their performance at a music school or at home. Furthermore, by displaying the key on / off speed for each note, the quality of each note's performance can be visually assessed. The key-on and key-off speeds for each note of the performance can also be displayed for reference. With the above configuration, the performance state of the performer Ua while playing a piece of music can be visually and intuitively grasped.

[0152] On the reference screen G9 of FIG. 25 , by tapping or otherwise inputting a portion where staccato playing is indicated, the music data for that portion can be displayed. The reference screen G9 of FIG. 25 shows the sound production status of each note, with the vertical axis representing volume and the horizontal axis representing time. The "Model Performance" in FIG. 25 shows the waveform of a model performance, which is data from a performance by a music teacher in a music class, etc., while the "Student Performance" in FIG. 25 shows a performance by performer Ua. In the "Model Performance," silent periods (beats) are formed between each note, indicating that staccato playing is being performed. On the other hand, in the "Student Performance," there is a connection between each note, indicating that staccato playing is not being performed well.

[0153] The performance analysis unit 52 can analyze the performance of the performer Ua and display the detected performance characteristics F on the musical score using marks or other means. The performer Ua can recognize that the performance is poor because there are no marks where musical symbols should be. It can also display detailed data in a graph, as shown in the figure, and output comments depending on the situation. For example, if the analysis results show that the performer Ua's touch on the keyboard is shallow and the keystroke speed is slow, it can output a comment such as, "Try to strike the keys a little faster and touch the keys all the way to the bottom." The selected music data can be selected not only from model performances, but also from past performance data from the performer Ua's music classes, performances at recitals, and other data, selected by the performer Ua.

[0154] By tapping or otherwise inputting a point where pianissimo playing is indicated, the music data for that point can be displayed. Figure 26 shows the relationship between key pressing speed and key depth for a single note, and the relationship between the key pressing speed of consecutive notes, normalized to zero as the average key pressing speed for the performance. Although pianissimo is indicated in the second measure, the key pressing speed for the single note is close to the appropriate value for pp in the figure, and the key pressing speed for consecutive notes is also lower than average. The performance analysis unit 52 can analyze the performance of performer Ua and display the detected performance characteristic F on the musical score G91 using a mark or other indicator. Performer Ua can recognize that pianissimo playing is being performed because a mark with "Good!" is displayed at the location of the musical symbol.

[0155] By tapping or otherwise inputting the location where the key release speed is displayed, the music data for that location can be displayed. In general, key release speed, which represents the speed at which a pressed key 182 returns to its original position, is an important factor in expressing a performance. The "Key Release Speed!" caution symbol is a performance expression instruction not included in the musical score G91, but based on the experience of piano teachers at music schools and the results of analyzing musical data, the caution symbol can be inserted in locations where key release speed is particularly slow and affects performance expression. The performance analysis unit 52 can analyze the performance of the performer Ua and display the detected performance characteristic F using marks or other indicators on the musical score. The performer Ua can recognize locations where the key release speed is slow and affects performance expression because the "Key Release Speed!" caution symbol is displayed. In addition, detailed data such as the one shown in the figure can be displayed in a graph, and comments can be output depending on the situation. For example, for the 13th note, which displays a warning about keystroke speed, if the keystroke speed is slow and the measurement of key depth shows that there is more contact than necessary after the attack point where the sound is produced, a comment such as "Your key release speed seems slow. Speed ​​up your keystroke speed a little to minimize contact at the attack point. If you release your fingers from the key in the direction of your grip, you will be able to release the key properly" can be output. Comments based on the results of data analysis can be displayed by collecting instruction given in regular music classes and linking them based on data analysis.

[0156] (2) The items displayed on the reference screen G can be specified by the user U. The item specification screen allows selection and setting for each item, such as performance characteristics, performance expression, and biometric information. In addition, by specifying a piece of music to be played, data can be collected during the period in which a specific piece of music is being played and the evaluation results can be displayed. Information collected by the terminal device 30, such as location and equipment selection, can be analyzed, sorted, and displayed to convey the items the user U wants to know.

[0157] (3) As described above, the terminal device 30 (characteristics acquisition unit 51) can acquire the performance characteristics Xa_A and tuning information Xb_A of the keyboard instrument 10A, and the performance characteristics Xa_B and tuning information Xb_B of the keyboard instrument 10B. The terminal device 30 may apply the performance characteristics Xa_A and tuning information Xb_A of the keyboard instrument 10A to the keyboard instrument 10B.

[0158] For example, the control device 31 of the terminal device 30 logs in to the keyboard instrument 10B and transmits the motion characteristic Xa_A and the tuning information Xb_A to the keyboard instrument 10B from the communication device 33. The control device 11 of the keyboard instrument 10B replaces the motion characteristic Xa_B and the tuning information Xb_B stored in the storage device 12 with the motion characteristic Xa_A and the tuning information Xb_A received from the terminal device 30.

[0159] With the above configuration, the performance characteristic Xa_B of the keyboard instrument 10B can be adjusted to be the same as the performance characteristic Xa_B of the keyboard instrument 10A. For example, it is possible to apply the performance characteristic Xa and tuning information Xb of a keyboard instrument 10 installed at a concert venue to a keyboard instrument 10 installed at home. Note that while the above explanation assumes that the performance characteristic Xa_A and tuning information Xb_A are applied to the keyboard instrument 10B, the performance characteristic Xa_B and tuning information Xb_B of the keyboard instrument 10B may also be applied to the keyboard instrument 10A.

[0160] (4) The operation of the keyboard instrument 10 may be controlled in response to an instruction from the performer Ua on the reference screen G. For example, Fig. 27 illustrates a reference screen G4 that includes a comparison image E (EA, EB) and a comment C, similar to the reference screen G4 illustrated in Fig. 17. The display control unit 53 displays an operation button G41 on the reference screen G4 for selecting whether or not to change the settings of the keyboard instrument 10 in accordance with the comment C.

[0161] The performer Ua instructs the keyboard instrument 10 to change or maintain the settings by operating the operating device 35. The terminal device 30 controls the settings (e.g., the motion characteristic Xa) of the keyboard instrument 10 in response to the instruction from the performer Ua. For example, the control device 31 instructs the keyboard instrument 10 to change the motion characteristic Xa. The control device 11 of the keyboard instrument 10 changes the motion characteristic Xa in response to the instruction from the terminal device 30. For example, when an instruction to change the touch feeling is given as shown in the example of FIG. 27 , the control device 11 changes the key depression sensitivity of the motion characteristic Xa.

[0162] 28 shows an example of a reference screen G5 including an evaluation image G51 and a comment C, similar to the reference screen G5 shown in Fig. 18. The display control unit 53 displays an operation button G52 on the reference screen G5 for selecting whether or not to change the operation mode of the keyboard instrument 10 in accordance with the comment C.

[0163] The performer Ua instructs the keyboard instrument 10 to change / maintain the operation mode by operating the operation device 35. The terminal device 30 controls the operation mode of the keyboard instrument 10 in response to the instruction from the performer Ua. For example, the control device 31 instructs the keyboard instrument 10 to change the operation mode. The control device 11 of the keyboard instrument 10 changes the operation mode to the acoustic mode in response to the instruction from the terminal device 30.

[0164] (5) In the above-described embodiments, the comment C stored in the storage device 32 is displayed on the display device 34. However, the configuration and method for displaying the comment C on the display device 34 are not limited to the above examples. For example, the comment C may be generated using artificial intelligence technology. Specifically, a machine-learned generative model is used to generate the comment C.

[0165] For example, the generative model used in the first to third embodiments is a statistical model that learns the relationship between a set of a movement characteristic Xa and a performance characteristic F and a comment C. By processing the set of a movement characteristic Xa and a performance characteristic F, the generative model generates a comment C that is statistically valid for the set.

[0166] The generative model used in the fourth and fifth embodiments is a statistical model that learns the relationship between a pair of performance characteristics F_A and F_B and a comment C. The generative model processes the pair of performance characteristics F_A and F_B to generate a comment C that is statistically appropriate for the pair.

[0167] The generative models exemplified above are interactive large-scale language models (LLMs) trained specifically for natural language processing tasks such as response generation. For example, a natural language processing model implemented by a Transformer model using a self-attention mechanism is an example of a generative model.

[0168] (6) In the above-described embodiments, the performance analysis unit 52 identifies the performance characteristic F by using the history information Xc and the sound pickup signal. However, the method of identifying the performance characteristic F is not limited to the above examples. For example, the performance analysis unit 52 may identify the performance characteristic F by using only the sound pickup signal. In other words, the use of the history information Xc may be omitted in identifying the performance characteristic F. Therefore, the configuration for acquiring the history information Xc from the keyboard instrument 10 (the characteristic acquisition unit 51) may be omitted.

[0169] (7) In each of the above-described embodiments, the terminal device 30 owned by the user U realizes each element of FIG. 10 . However, some or all of the elements illustrated in FIG. 10 (the characteristic acquisition unit 51, the performance analysis unit 52, and the display control unit 53) may be realized by a server device (e.g., a cloud server) that communicates with the terminal device 30.

[0170] (8) In each of the above-described embodiments, a keyboard instrument 10 equipped with both a sound generation mechanism 183 and a sound source device 16 has been exemplified. However, the present disclosure also applies to a natural keyboard instrument 10 that does not have a sound source device 16 and produces performance sounds using a sound generation mechanism 183, or to an electronic keyboard instrument 10 that does not have a sound generation mechanism 183 and produces performance sounds electronically using a sound source device 16.

[0171] The instrument played by the performer Ua is not limited to the keyboard instrument 10. In other words, the present disclosure can be applied to managing the performance of any type of instrument. Furthermore, an instrument-type input device (e.g., a MIDI keyboard) that is not equipped with a function for generating performance sounds may be substituted for the keyboard instrument 10 in each of the above-described embodiments. As can be understood from the above explanation, the "performance device" in this disclosure is a comprehensive expression of a device used by a user (e.g., the performer Ua) to play music. The keyboard instrument 10 illustrated in each of the above-described embodiments is an example of a performance device.

[0172] (9) In each of the above-described embodiments, the information processing system 100 includes a keyboard instrument 10. However, the keyboard instrument 10 may be omitted from the information processing system 100. For example, the keyboard instrument 10 may be installed in an acoustic space located remotely from the information processing system 100 (terminal device 30). Information generated by the keyboard instrument 10 is transmitted to the information processing system 100 via a communication network such as the Internet. The operation of the terminal device 30 is the same as in each of the above-described embodiments. As can be understood from the above description, the "information processing system" in this disclosure may or may not include a musical instrument. Therefore, the terminal device 30 alone in each of the above-described embodiments is also included in the "information processing system."

[0173] (10) As mentioned above, the functions of the terminal device 30 exemplified above are realized through cooperation between one or more processors constituting the control device 31 and a program stored in the storage device 32. The program according to the present disclosure may be provided in a form stored on a computer-readable recording medium and installed on a computer. The recording medium may be, for example, a non-transitory recording medium, such as an optical recording medium (optical disk) such as a CD-ROM, but may also include any known form of recording medium, such as a semiconductor recording medium or a magnetic recording medium. Note that a non-transitory recording medium includes any recording medium other than a transient, propagating signal, and does not exclude volatile recording media. Furthermore, in a configuration in which a distribution device distributes a program via a communication network, the storage medium that stores the program in the distribution device corresponds to the non-transitory recording medium described above.

[0174] J: Supplementary Note From the above-described exemplary embodiments, the following configurations can be understood, for example.

[0175] According to one aspect (aspect 1) of the present disclosure, an information processing system includes a motion characteristic acquisition unit that acquires a first motion characteristic related to a first musical performance device, a performance analysis unit that identifies a first performance characteristic corresponding to the first motion characteristic by analyzing a performance result of a user using the first musical performance device, and a display control unit that displays the first motion characteristic and the first performance characteristic on a display device. According to this aspect, the first motion characteristic related to the first musical performance device and the second performance characteristic related to a performance using the first musical performance device are displayed, allowing the user to visually and intuitively grasp the relationship of their own performance characteristic to the motion characteristic of the first musical performance device.

[0176] The first movement characteristic and the first performance characteristic may be displayed in any manner. Specifically, the first movement characteristic and the first performance characteristic may be displayed in a manner that allows a user to compare them. For example, the first movement characteristic and the first performance characteristic may be displayed in parallel, or the first movement characteristic and the second performance characteristic may be displayed alternately.

[0177] In a specific example (Aspect 2) of Aspect 1, the performance analysis unit identifies the first performance characteristic for each of a plurality of time periods, and the display control unit displays the first action characteristic and the first performance characteristic for each of the plurality of time periods. According to this aspect, the first performance characteristic for each of the plurality of time periods is displayed, allowing a user to visually confirm temporal fluctuations of the first performance characteristic.

[0178] Each of the "multiple periods" is a period of a predetermined length on a time axis. For example, the seven days that make up a week are an example of the "multiple periods." The time lengths of the periods may be the same or different. For example, "weekdays (Monday to Friday)" and "weekends (Saturday and Sunday)" may be considered "multiple periods." Furthermore, each of the multiple periods may partially overlap.

[0179] In a specific example (Aspect 3) of Aspect 1 or Aspect 2, the behavior characteristic acquisition unit further acquires second behavior characteristics for a second performance device having characteristics different from those of the first performance device, the performance analysis unit further identifies second performance characteristics corresponding to the second behavior characteristics by analyzing the results of the user's performance using the second performance device, and the display control unit displays the relationship between the first behavior characteristics and the first performance characteristics and the relationship between the second behavior characteristics and the second performance characteristics. According to the above aspect, the user can visually compare the first behavior characteristics of the first performance device with the second behavior characteristics of the second performance device, as well as the first performance characteristics when playing the first performance device with the second performance characteristics when playing the second performance device. This allows the user to easily understand points to note when playing each of the first and second performance devices.

[0180] The "second performance device" is a performance device with different characteristics from the first performance device. For example, the first performance device and the second performance device are different types of musical instruments. For example, the first performance device is a grand piano and the second performance device is an upright piano. The first performance device and the second performance device may also be the same type of musical instrument but different models. For example, the first performance device and the second performance device may be different types of electronic keyboard instruments. Furthermore, the first performance device and the second performance device may have a common type and model but different characteristics. For example, the first performance device and the second performance device may have a common type and model but different characteristics as a result of various settings.

[0181] According to one aspect (aspect 4) of the present disclosure, an information processing system includes a performance analysis unit that identifies a first performance characteristic by analyzing a performance performed by a user using a first performance device and identifies a second performance characteristic by analyzing a performance performed by the user using a second performance device having characteristics different from those of the first performance device, and a display control unit that displays the first performance characteristic and the second performance characteristic on a display device. According to this aspect, the first performance characteristic related to the performance performed using the first performance device and the second performance characteristic related to the performance performed using the second performance device are displayed, allowing the user to visually and intuitively grasp the changes in the performance characteristics resulting from the differences between the performance devices.

[0182] In a specific example (Aspect 5) of Aspect 4, the first and second performance characteristics each include an evaluation value for each of a plurality of evaluation items, and the display control unit displays the evaluation value for each of the first and second performance characteristics for each of the evaluation items. According to this aspect, the evaluation value for the first and second performance characteristics is displayed for each of a plurality of evaluation items related to performance. Therefore, a user can grasp changes in performance characteristics due to differences in performance devices from multiple perspectives (evaluation items).

[0183] An information processing method according to one aspect (aspect 7) of the present disclosure acquires a first operating characteristic related to a first performance device, analyzes the results of a user's performance using the first performance device, identifies a first performance characteristic corresponding to the first operating characteristic, and displays the first operating characteristic and the first performance characteristic on a display device.

[0184] A program according to one aspect (aspect 8) of the present disclosure causes a computer system to function as: a motion characteristic acquisition unit that acquires a first motion characteristic related to a first performance device; a performance analysis unit that identifies a first performance characteristic corresponding to the first motion characteristic by analyzing the results of a performance performed by a user using the first performance device; and a display control unit that displays the first motion characteristic and the first performance characteristic on a display device.

[0185] 100...information processing system, 10...keyboard instrument, 11...control device, 12...storage device, 13...communication device, 14...display device, 15...operation device, 16...sound source device, 17...sound emission device, 18...performance mechanism, 181...keyboard, 182...key, 183...sound generation mechanism, 184...pedal, 185...detection device, 20...recording system, 21...imaging device, 22...sound collection device, 30 (30a, 30b, 30c)...terminal device, 31...control device, 32...storage device, 33...communication device, 3 4...display device, 35...operation device, 36...sound emission device, 37...sound collection device, 40...measurement system, 41...biometric sensor, 42...center of gravity sensor, 51...characteristic acquisition unit, 52...performance analysis unit, 53...display control unit, 55...music analysis unit, 56...video editing unit, 57...playback control unit, 60...setting screen, 61...guidance screen, 62...device icon, 62a...imaging icon, 62b...sound collection icon, 63...scene selection window, 64...device setting window, 90...recording system.

Claims

1. An information processing system comprising: an operation characteristic acquisition unit that acquires a first operation characteristic related to a first performance device; a performance analysis unit that specifies a first performance characteristic corresponding to the first operation characteristic by analyzing a result of a performance by a user using the first performance device; and a display control unit that displays the first operation characteristic and the first performance characteristic on a display device.

2. The information processing system according to claim 1, wherein the performance analysis unit specifies the first performance characteristic for each of a plurality of periods, and the display control unit displays the first operation characteristic and the first performance characteristic in each of the plurality of periods.

3. The information processing system according to claim 1, wherein the operation characteristic acquisition unit further acquires a second operation characteristic related to a second performance device having characteristics different from those of the first performance device, the performance analysis unit further specifies a second performance characteristic corresponding to the second operation characteristic by analyzing a result of a performance by the user using the second performance device, and the display control unit displays a relationship between the first operation characteristic and the first performance characteristic and a relationship between the second operation characteristic and the second performance characteristic.

4. An information processing system comprising: a performance analysis unit that specifies a first performance characteristic by analyzing a result of a performance by a user using a first performance device, and specifies a second performance characteristic by analyzing a result of a performance by the user using a second performance device having characteristics different from those of the first performance device; and a display control unit that displays the first performance characteristic and the second performance characteristic on a display device.

5. The information processing system according to claim 4, wherein each of the first performance characteristic and the second performance characteristic includes an evaluation value related to each of a plurality of evaluation items, and the display control unit displays the evaluation values in each of the first performance characteristic and the second performance characteristic for each of the evaluation items.

6. An information processing method realized by a computer system, the method comprising: acquiring a first operation characteristic related to a first performance device; specifying a first performance characteristic corresponding to the first operation characteristic by analyzing a result of a performance by a user using the first performance device; and displaying the first operation characteristic and the first performance characteristic on a display device.

7. A program that causes a computer system to function as an operation characteristic acquisition unit that acquires first operation characteristics related to a first performance device, a performance analysis unit that identifies first performance characteristics corresponding to the first operation characteristics by analyzing the results of performance by a user using the first performance device, and a display control unit that displays the first operation characteristics and the first performance characteristics on a display device.

Images