Information processing system, method for processing information, and program

Abstract

To allow a user to visually and intuitively get a schematic relation on a time axis between a playing time and a speaking time.SOLUTION: An information processing system includes: a recording data acquisition unit 511 for acquiring performance data showing a performance that a performer does with a musical instrument and voice data showing speech voice regarding a performance of the performer; and a display control unit 514 for displaying, in a display device, a reference screen showing at least one performance time in which the performance of the performance data exists and at least one speech time in which speech voice of the voice data exists, on the same time axis.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates to technology for assisting musicians in practicing musical instruments. [Background technology]

[0002] Various technologies for supporting musical instrument practice have been proposed. For example, Patent Document 1 discloses a technology for supporting playing by providing feedback to a user on information based on the degree of coordination of multiple motor elements in playing the piano. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2019 / 130755 Summary of the Invention [Problem to be solved by the invention]

[0004] In a situation where a performer is practicing an instrument, for example, if the performer can grasp the temporal relationship between the period when the performer is actually playing the instrument and the period when the performer is conversing with, for example, an instructor, the efficiency of the instrument practice can be expected to be further improved. In consideration of the above circumstances, one aspect of the present disclosure aims to enable a user to visually and intuitively grasp the general relationship on the time axis between the performance period and the speaking period. [Means for solving the problem]

[0005] In order to solve the above problems, an information processing system according to one aspect of the present disclosure includes a recorded data acquisition unit that acquires performance data representing a performance of an instrument by a performer and audio data representing speech related to the performance by the performer, and a display control unit that displays on a display device a reference screen that represents, on a common time axis, one or more performance periods in which the performance represented by the performance data exists and one or more speech periods in which the speech represented by the audio data exists.

[0006] An information processing method according to one aspect of the present disclosure acquires performance data representing a performance of an instrument by a performer and audio data representing speech related to the performance by the performer, and displays on a display device a reference screen that shows, on a common time axis, one or more performance periods in which the performance represented by the performance data exists and one or more speech periods in which the speech represented by the audio data exists.

[0007] A program according to one aspect of the present disclosure causes a computer system to function as a recorded data acquisition unit that acquires performance data representing a performance of an instrument by a performer and audio data representing speech related to the performance by the performer, and a display control unit that displays on a display device a reference screen that represents, on a common time axis, one or more performance periods in which the performance represented by the performance data exists and one or more speech periods in which the speech represented by the audio data exists. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram illustrating a configuration of an information processing system according to a first embodiment. [Figure 2] FIG. 2 is a block diagram illustrating a configuration of a terminal device. [Figure 3] FIG. 1 is a block diagram illustrating the configuration of a keyboard instrument. [Figure 4] FIG. 1 is a block diagram illustrating a configuration of a control system. [Figure 5] FIG. 2 is a block diagram illustrating an example of the functional configuration of the control system. [Figure 6] 10 is a flowchart of a process executed by a recorded data acquisition unit. [Figure 7] FIG. 1 is a schematic diagram of recorded data. [Figure 8] FIG. 2 is a schematic diagram of management data. [Figure 9] FIG. 10 is a schematic diagram of a reference screen and detailed information. [Figure 10] 10 is a flowchart of a control process. [Figure 11]FIG. 2 is a schematic diagram of an image displayed on a display device of the terminal device. [Figure 12] 10A and 10B are schematic diagrams of a reference screen and detailed information in the second embodiment. [Figure 13] 13A and 13B are schematic diagrams of a reference screen and detailed information in the third embodiment. [Figure 14] FIG. 13 is a schematic diagram of an image displayed on a display device of a terminal device in the fourth embodiment. [Figure 15] FIG. 13 is a schematic diagram of an image displayed on a display device of a terminal device in the fifth embodiment. [Figure 16] FIG. 20 is a schematic diagram of a reference screen in the sixth embodiment. [Figure 17] FIG. 20 is a schematic diagram of an image displayed on a display device of a terminal device in the seventh embodiment. [Figure 18] FIG. 20 is a schematic diagram of an image displayed on a display device of a terminal device in the seventh embodiment. [Figure 19] FIG. 13 is a schematic diagram of a reference screen in the seventh embodiment; [Figure 20] FIG. 20 is a schematic diagram of an image displayed on a display device of a terminal device in the eighth embodiment. [Figure 21] FIG. 13 is a schematic diagram of an image displayed on a display device of a terminal device in the ninth embodiment. [Figure 22] FIG. 13 is a schematic diagram of an image displayed on a display device of a terminal device in the ninth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] A: First embodiment FIG. 1 is a block diagram illustrating the configuration of an information processing system 100 according to a first embodiment. The information processing system 100 is a computer system (i.e., a performance recording system or a performance management system) for recording and managing a performance of a keyboard instrument 10 by a performer Ua. The performer Ua is, for example, a person who practices playing the keyboard instrument 10. The performer Ua is, for example, a student belonging to a music school. In addition to the performer Ua, an instructor Ub is also present at the music school. The instructor Ub instructs the performer Ua on playing the keyboard instrument 10.

[0010] The information processing system 100 includes a keyboard instrument 10, a recording system 20, a terminal device 30, a measurement system 40, and a control system 50. The terminal device 30 is an information device such as a smartphone, a tablet terminal, or a personal computer.

[0011] 2 is a block diagram illustrating the configuration of the terminal device 30. The terminal device 30 includes a control device 31, a storage device 32, a communication device 33, a display device 34, an operation device 35, a sound emitting device 36, and a sound collecting device 37. The terminal device 30 may be realized as a single device, or may be realized as a plurality of devices configured separately from each other.

[0012] The control device 31 is composed of one or more processors that control each element of the terminal device 30. For example, the control device 31 is composed of one or more types of processors such as a central processing unit (CPU), a graphics processing unit (GPU), a sound processing unit (SPU), a digital signal processor (DSP), a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC).

[0013] The storage device 32 is one or more memories that store programs executed by the control device 31 and various data used by the control device 31. The storage device 32 is configured with a known storage medium such as a magnetic storage medium or a semiconductor storage medium. The storage device 32 may be configured with a combination of multiple types of storage media. Furthermore, a portable storage medium that is detachable from the terminal device 30, or a storage medium (e.g., cloud storage) that the control device 31 can write to or read from via the communication network 200 may be used as the storage device 32.

[0014] The communication device 33 communicates with peripheral devices (e.g., the keyboard instrument 10, the recording system 20, the measurement system 40, and the control system 50). The communication device 33 communicates with the peripheral devices via wireless communication, such as Wi-Fi (registered trademark) or Bluetooth (registered trademark). The communication between the communication device 33 and the peripheral devices may be wired communication. The communication device 33 communicates with the control system 50 via a communication network 200, such as the Internet.

[0015] The display device 34 displays images under the control of the control device 31. The display device 34 is configured with a display panel such as a liquid crystal panel or an organic EL (Electroluminescence) panel. The operation device 35 is an input device that accepts operations by the user U (performer Ua or instructor Ub) of the terminal device 30. For example, an operator operated by the user U or a touch panel that detects contact by the user U is used as the operation device 35. Note that the display device 34 or the operation device 35, which are separate from the terminal device 30, may be connected to the terminal device 30 by wire or wirelessly. Note that the user U is not limited to the performer Ua and the instructor Ub. For example, the user U may also include the guardian of the performer Ua.

[0016] The sound emitting device 36 emits sound waves under the control of the control device 31. The sound emitting device 36 is an output device such as a speaker or headphones. The sound emitting device 36 may include a D / A converter that converts the acoustic signal of the sound emitting target from digital to analog, and an amplifier that amplifies the acoustic signal. Note that the sound emitting device 36, which is separate from the terminal device 30, may be connected to the terminal device 30 by wire or wirelessly.

[0017] The sound collection device 37 collects ambient sounds to generate an acoustic signal (hereinafter referred to as a "collected sound signal"). Specifically, the sound collection device 37 collects the performance sound emitted from the keyboard instrument 10 by the performer Ua. In other words, the collected sound signal is a signal representing the waveform of the performance sound of the keyboard instrument 10. For example, the sound collection device 37 may be a single microphone or a microphone array in which multiple microphones are arranged. For convenience, an A / D converter that converts the collected sound signal from analog to digital and an amplifier that amplifies the collected sound signal are not shown in the figure. The sound collection device 37 may be separate from the terminal device 30 and connected to the terminal device 30 by wire or wirelessly. The control device 31 stores the collected sound signal generated by the sound collection device 37 in the storage device 32.

[0018] 3 is a block diagram illustrating the configuration of a keyboard instrument 10. The keyboard instrument 10 is an instrument that accepts a performance by a performer Ua, and includes a control device 11, a storage device 12, a communication device 13, a display device 14, an operation device 15, a sound source device 16, a sound output device 17, and a performance mechanism 18. The keyboard instrument 10 is an example of a performance device that the performer Ua uses for performance. Note that some or all of the functions of the terminal device 30 may be incorporated into the keyboard instrument 10.

[0019] The performance mechanism 18 includes a keyboard 181, a sound generation mechanism 183, pedals 184, and a detection device 185. The keyboard 181 is a mechanism in which a plurality of keys 182, including white keys and black keys, are arranged. Each of the plurality of keys 182 is a performance operator that receives operation (e.g., key depression and key release) by the performer Ua. The sound generation mechanism 183 is an action mechanism that generates performance sounds by vibrating strings in response to operation of each key 182 on the keyboard 181. Specifically, the sound generation mechanism 183 includes strings, which are an example of a sound source, hammers that rotate to strike the strings, and a transmission mechanism (e.g., a wippen, jack, or repetition lever) that rotates the hammer in response to the displacement of each key 182. The sound generation mechanism 183 also includes a mute mechanism that prevents the strings from being struck in mute mode.

[0020] The pedal 184 is a performance operator that the performer Ua operates with his / her foot. For example, the pedal 184 may be a damper pedal (sustain pedal) for instructing the extension of a performance sound, or a soft pedal for reducing the volume of a performance sound.

[0021] The detection device 185 is a sensor that detects the operation of each key 182 and pedal 184. Specifically, the detection device 185 detects the amount of displacement of each key 182 and pedal 184. The detection device 185 is configured, for example, by an optical, mechanical, or magnetic sensor.

[0022] The control device 11 is composed of one or more processors that control each element of the keyboard instrument 10. For example, the control device 11 is composed of one or more types of processors such as a CPU, a GPU, an SPU, a DSP, an FPGA, or an ASIC.

[0023] The control device 11 generates performance data D in accordance with the detection result by the detection device 185. The performance data D is music data representing the performance of the keyboard instrument 10 by the performer Ua. Specifically, the performance data D is a time series of event data conforming to, for example, the MIDI (Musical Instrument Digital Interface) standard. That is, the performance data D specifies, in time series, the pitches (note numbers) corresponding to the keys 182 operated by the performer Ua and the intensity (velocity) of the operation. The performance data D also represents the operation of the pedal 184 by the performer Ua. As described above, the control device 11 functions as a music data generation unit that generates performance data D in accordance with the detection result by the detection device 185.

[0024] The display device 14 displays an image under the control of the control device 31. The display device 14 is configured with a display panel such as a liquid crystal panel or an organic EL panel. The operation device 15 is an input device that accepts operations by the performer Ua. For example, an operator operated by the performer Ua or a touch panel that detects contact by the performer Ua is used as the operation device 15. Note that the display device 14 or the operation device 15, which are separate from the keyboard instrument 10, may be connected to the keyboard instrument 10 by wire or wirelessly.

[0025] The sound source device 16 generates an audio signal representing a performance sound according to the result of detection by the detection device 185. Specifically, the sound source device 16 generates an audio signal according to the performance data D generated by the control device 11. For example, an audio signal of a pitch corresponding to the key 182 whose displacement has been detected by the detection device 185 is generated. Note that the control device 11 may realize the function of the sound source device 16 by executing a program stored in the storage device 12. As described above, the keyboard instrument 10 is an electronic musical instrument that can selectively generate natural sounds by the sound generation mechanism 183 and electronic sounds by the sound source device 16. Note that one of the sound generation mechanism 183 and the sound source device 16 may be omitted.

[0026] The sound emitting device 17 emits sound waves under the control of the control device 11. The sound emitting device 17 is an output device such as a speaker or headphones. Specifically, the sound emitting device 17 emits a performance sound represented by an acoustic signal generated by the sound source device 16. As described above, the keyboard instrument 10 of the first embodiment can selectively generate a performance sound by the sound generation mechanism 183 or by the sound source device 16. The sound emitting device 17 may include a D / A converter that converts the acoustic signal from digital to analog, and an amplifier that amplifies the acoustic signal. The sound emitting device 17 may be separate from the keyboard instrument 10 and connected to the keyboard instrument 10 by wire or wirelessly.

[0027] The storage device 12 is one or more memories that store programs executed by the control device 11 and various data used by the control device 11. The storage device 12 is configured with a known storage medium such as a magnetic storage medium or a semiconductor storage medium. The storage device 12 may also be configured with a combination of multiple types of storage medium. Furthermore, the storage device 12 may be a portable storage medium that is detachable from the keyboard instrument 10, or a storage medium (e.g., cloud storage) that the control device 11 can write to or read from via the communication network 200.

[0028] The storage device 12 also stores performance information X. The performance information X is information related to the performance by the performer Ua, and is temporarily stored in the keyboard instrument 10 until it is transferred to the terminal device 30. In other words, after being transferred to the terminal device 30, the performance information X is deleted from the storage device 12. The performance information X can also be expressed as information (performance log) that represents the performance history of the performer Ua. Specifically, the performance information X includes the identification information of the performer Ua, the sound source used, the mode used, the performance date and time (start time and end time), the performance location, the performance scene, the identification information and model of the keyboard instrument 10, the name of the training score used by the performer Ua, and score information of the composition performed by the performer Ua. The performance information X also includes the aforementioned performance data D that represents the performance by the performer Ua.

[0029] The communication device 13 communicates with the terminal device 30. The communication device 13 communicates with the terminal device 30 via wireless communication such as Wi-Fi (registered trademark) or Bluetooth (registered trademark). For example, the communication device 13 transmits performance information X stored in the storage device 12 to the terminal device 30. Note that the communication between the communication device 13 and the terminal device 30 may be wired communication. The communication device 13, which is separate from the keyboard instrument 10, may be connected to the keyboard instrument 10 via wire or wirelessly.

[0030] 1 records a performance by a performer Ua. Specifically, the recording system 20 includes an imaging device 21 and a sound collection device 22 installed in a music classroom. The imaging device 21 may be mounted on the terminal device 30 or the keyboard instrument 10. Similarly, the sound collection device 22 may be mounted on the terminal device 30 or the keyboard instrument 10. The sound collection device 22 may be mounted on the imaging device 21. The recording system 20 may include multiple imaging devices 21 or multiple sound collection devices 22.

[0031] The imaging device 21 is a video camera that captures images of the performer Ua and the keyboard instrument 10 to generate video data V. The video data V is data of moving images captured of the performer Ua playing the keyboard instrument 10. The video data V may be in any format, for example, MPEG (moving picture experts group) format. Specifically, the imaging device 21 includes multiple video cameras that capture images of the performer Ua from different directions. Therefore, the video data V includes multiple moving images captured of the performer Ua playing the keyboard instrument 10 from different directions (see, for example, "partial video E3" in FIG. 12).

[0032] The imaging device 21 includes, for example, an optical system such as a photographing lens, an imaging element that receives incident light from the optical system, and a processing circuit that generates video data V according to the amount of light received by the imaging element. In addition to video equipment dedicated to imaging, a general-purpose information device (e.g., a smartphone or tablet terminal) equipped with an imaging function may also be used as the imaging device 21. The imaging device 21 may start capturing images in conjunction with the start of performance by the performer Ua on the keyboard instrument 10.

[0033] 1 is one or more microphones that collect sounds around the performer Ua. The sound collection device 22 generates audio data A that represents the waveform of the surrounding sounds. The audio data A may be in any format, but may be in wav format, for example.

[0034] Specifically, the sound collection device 22 records not only the performance sounds emitted from the keyboard instrument 10 by the performer Ua, but also the voice spoken by the instructor Ub (hereinafter referred to as "spoken voice"). The spoken voice of the instructor Ub is voice related to the performance by the performer Ua. Examples of spoken voice include various voices used to instruct the performer Ua on the performance of the keyboard instrument 10, such as explaining how to play, evaluating the skill of the performance, and pointing out problems with the performance.

[0035] The sound collection device 37 of the terminal device 30 may generate the audio data A of the spoken voice in the same manner as the sound collection device 22. In addition to a device dedicated to sound collection, a general-purpose information device (for example, a smartphone or tablet terminal) equipped with a sound collection function may be used as the sound collection device 22. Sound collection by the sound collection device 22 may start in conjunction with the start of performance of the keyboard instrument 10 by the performer Ua.

[0036] The imaging device 21 is connected to the terminal device 30 by wire or wirelessly. Video data V generated by the imaging device 21 is transmitted to the terminal device 30. Similarly, the sound collection device 22 is connected to the terminal device 30 by wire or wirelessly. Audio data A generated by the sound collection device 22 is transmitted to the terminal device 30.

[0037] The measurement system 40 in FIG. 1 measures biometric information Y related to the player Ua. Specifically, the measurement system 40 includes a biometric sensor 41 and a center-of-gravity sensor 42. The biometric sensor 41 is a portable sensor worn on the body of the player Ua. The biometric sensor 41 is, for example, a heart rate sensor that detects the heart rate of the player Ua. The center-of-gravity sensor 42 is a sensor installed on the chair on which the player Ua sits while playing the keyboard instrument 10, and detects movement of the center of gravity of the seat. The biometric information Y of the player Ua is data including, for example, the playing posture, tension (strain), heart rate, and concentration of the player Ua.

[0038] The playing posture is an index relating to the posture of the player Ua playing the keyboard instrument 10. The playing posture is determined by analyzing the detection results of the center of gravity sensor 42. Note that the playing posture may also be determined by analyzing an image of the player Ua captured by the imaging device 21.

[0039] The heart rate is detected, for example, by a heart rate sensor. The tension level is an index of the player Ua's level of tension (strain) and is calculated, for example, based on the player Ua's heart rate and the stability of his / her playing posture. The concentration level is an index of the player Ua's level of concentration on playing and is calculated, for example, based on the player Ua's heart rate and the stability of his / her playing posture. Note that the concentration level may be set to a larger value the more times a key is pressed within a specified time. Note that the configuration and type of the measurement system 40 are not limited to the above examples.

[0040] The configuration for acquiring the biometric information Y is not limited to the portable biometric sensor 41 attached to the body of the performer Ua. For example, the biometric information Y of the performer Ua may be generated by analyzing video data V generated by imaging the performer Ua using video equipment such as the imaging device 21. To generate the biometric information Y by analyzing the video data V, for example, the technology described in JP 2021-069813 A is applied.

[0041] In the configuration described above, the terminal device 30 acquires various types of information from the peripheral devices (the keyboard instrument 10, the recording system 20, and the measurement system 40). For example, the control device 31 transmits an information request to the peripheral devices from the communication device 33, and receives, via the communication device 33, information transmitted from the peripheral devices in response to the information request. Specifically, the terminal device 30 acquires performance information X including performance data D from the keyboard instrument 10, acquires video data V and audio data A from the recording system 20, and acquires biometric information Y of the performer Ua from the measurement system 40.

[0042] Recorded data R1 is stored in the storage device 32 of the terminal device 30. The recorded data R1 is comprehensive data related to the performance by the performer Ua. Specifically, the recorded data R1 includes performance information X, video data V, audio data A, and biometric information Y. The recorded data R1 is stored in the storage device 32 every time the performer Ua practices (for example, every lesson in a music studio). The control device 31 transmits the recorded data R1 from the communication device 33 to the control system 50.

[0043] The control system 50 receives the recorded data R1 transmitted from the terminal device 30. The control system 50 is a computer system for managing the recorded data R1. Although the first embodiment illustrates a configuration in which the recorded data R1 is transmitted to the control system 50 via the terminal device 30, the recorded data R1 may be transmitted to the control system 50 without passing through the terminal device 30. For example, performance information X may be transmitted from the keyboard instrument 10 to the control system 50, video data V may be transmitted from the imaging device 21 to the control system 50, audio data A may be transmitted from the sound collection device 22 to the control system 50, and biometric information Y may be transmitted from the measurement system 40 to the control system 50.

[0044] 4 is a block diagram illustrating an example of the configuration of a control system 50. The control system 50 includes a control device 51, a storage device 52, and a communication device 53. The control system 50 may be realized as a single device, or may be realized as a plurality of devices configured separately from each other.

[0045] The control device 51 is composed of one or more processors that control each element of the keyboard instrument 10. For example, the control device 51 is composed of one or more types of processors such as a CPU, a GPU, an SPU, a DSP, an FPGA, or an ASIC.

[0046] The storage device 52 is one or more memories that store programs executed by the control device 51 and various data used by the control device 51. The storage device 52 is configured with a known storage medium such as a magnetic storage medium or a semiconductor storage medium. The storage device 52 may be configured with a combination of multiple types of storage media. Furthermore, a portable storage medium that is detachable from the control system 50, or a storage medium to which the control device 51 can write or read via the communication network 200 (for example, cloud storage) may be used as the storage device 52.

[0047] The communication device 53 communicates with the terminal device 30 via the communication network 200. Note that the communication between the communication device 53 and the terminal device 30 may be wireless communication such as Wi-Fi (registered trademark) or Bluetooth (registered trademark).

[0048] 5 is a block diagram illustrating an example of the functional configuration of the control system 50. The control device 51 executes a program stored in the storage device 52 to realize multiple functions for managing the recorded data R1 (a recorded data acquisition unit 511, an information processing unit 512, a performance evaluation unit 513, a display control unit 514, a selection receiving unit 515, and an information output unit 516).

[0049] The recorded data acquisition unit 511 acquires the recorded data R1. That is, the recorded data acquisition unit 511 acquires the performance information X (performance data D), the video data V, the audio data A, and the biometric information Y. Fig. 6 is a flowchart of the process executed by the recorded data acquisition unit 511. When the process starts, the recorded data acquisition unit 511 receives the recorded data R1 transmitted from the terminal device 30 via the communication device 53 (Sa1).

[0050] The recorded data acquisition unit 511 manages the temporal correspondence between the performance information X (performance data D), video data V, audio data A, and biometric information Y of the recorded data R1 (Sa2). Specifically, the recorded data acquisition unit 511 adds synchronization data to each of the performance data D, video data V, audio data A, and biometric information Y. The synchronization data represents a time point (hereinafter referred to as a "reference time point") corresponding in time between the performance data D, video data V, and audio data A. For example, LTC (Longitudinal Time Code) or MTC (MIDI Time Code) is exemplified as the synchronization data.

[0051] The time indicated by the synchronization data may be, for example, Coordinated Universal Time (UTC) or Japan Standard Time (JTC), or may be a time specific to the information processing system 100. When a specific time is used, the time is defined, for example, starting from the time when the terminal device 30 logs in to the control system 50.

[0052] Furthermore, synchronization data may be generated in peripheral devices of the terminal device 30 (the keyboard instrument 10, the image capture device 21, the sound collection device 22, and the measurement system 40). For example, the keyboard instrument 10 assigns MTC as synchronization data to the performance data D. The recorded data acquisition unit 511 assigns synchronization data common to the performance data D to data other than the performance data D (the video data V, the audio data A, and the biometric information Y). Furthermore, synchronization data may be shared between the peripheral devices.

[0053] The recorded data acquisition unit 511 generates recorded data R2 including the recorded data R1 (Sa3) and stores the recorded data R2 in the storage device 52 (Sa4). Note that the recorded data R1 and the recorded data R2 are examples of "recorded data" in this disclosure.

[0054] 7 is a schematic diagram of the recorded data R2. The recorded data R2 is data in which user identification information Fa, device identification information Fb, and information type Fc are added to each of the performance data D, video data V, and audio data A included in the recorded data R1.

[0055] The user identification information Fa is a code string for identifying the performer Ua who is the subject of recording in the recording data R1. For example, identification information included in the performance information X is registered as the user identification information Fa in the recording data R2. Note that, for example, the identification information of the terminal device 30 may be used as the user identification information Fa. Furthermore, the user identification information Fa may be changed in response to an instruction from the user U.

[0056] The device identification information Fb is a code for identifying the device used for recording. For example, the identification information of the keyboard instrument 10 ("Piano#1" in FIG. 7) is added to the performance data D as the device identification information Fb. The identification information of the imaging device 21 ("Cam#1" in FIG. 7) is added to the video data V as the device identification information Fb. The identification information of the sound collection device 22 ("Mic#1" in FIG. 7) is added to the audio data A as the device identification information Fb. The device identification information Fb may be included in the recording data R1 or may be generated by the recording data acquisition unit 511. The device identification information Fb may also be changed in response to an instruction from the user U.

[0057] The information type Fc is the type of information in each of the performance data D, video data V, and audio data A. For example, the information type Fc of "performance" is added to the performance data D, the information type Fc of "video" is added to the video data V, and the information type Fc of "audio" is added to the audio data A. The recorded data acquisition unit 511 recognizes the information type Fc, for example, by referring to the metadata added to each of the performance data D, video data V, and audio data A. Note that the information type Fc may be recognized using a recognition process that uses a statistical model such as a deep neural network or an HMM (Hidden Markov Model).

[0058] The information processing unit 512 in Figure 5 generates management data R3 from the recorded data R2. The management data R3 is data for managing the performance of the performer Ua over time. Figure 8 is a schematic diagram of the management data R3. The management data R3 is a collection of multiple unit data Z. Each of the multiple unit data Z corresponds to one of performance data D, video data V, and audio data A.

[0059] The performer Ua does not play the keyboard instrument 10 continuously, but plays intermittently at appropriate intervals. For example, the performer Ua stops playing when the instructor Ub speaks. The unit data Z corresponding to the performance data D is data for managing the period on the time axis during which the performance represented by the performance data D exists (hereinafter referred to as the "performance period"). A performance period is a period during which the performer Ua actually plays the keyboard instrument 10 (i.e., a period other than a period during which the performer Ua stops playing). For example, adjacent periods on the time axis separated by an interval greater than a predetermined length are divided into performance periods. In other words, intervals shorter than the predetermined length in the performance by the performer Ua are included in one performance period. Note that the predetermined length applied to the division of the performance period (i.e., a threshold value related to the interval between performance periods) may be a fixed length set in advance, or may be a variable length set arbitrarily by the user U, for example. Furthermore, if the duration of a performance period is less than a predetermined threshold (for example, 1 second), the information processing unit 512 may delete data within that performance period as noise data from the performance data D. In other words, data within an extremely short performance period from the performance data D may be excluded from storage in the storage device 52.

[0060] The unit data Z corresponding to the performance data D is generated for each performance period in the performance by the performer Ua. Therefore, the unit data Z is generated for the number of performance periods in the performance represented by the performance data D.

[0061] Each unit data Z corresponding to the performance data D includes user identification information Fa, device identification information Fb, and information type Fc, as well as period data T and performance data Qd (Qd1, Qd2, ...). The period data T is a timestamp specifying the performance period. Specifically, the period data T specifies the start time and end time of the performance period, starting from the reference point indicated by the synchronization data. The performance data Qd is the portion of the performance data D within the performance period specified by the period data T. In other words, the performance data Qd is time-series data representing the performance of the performer Ua within the performance period. As can be understood from the above explanation, each unit data Z includes performance data Qd obtained by dividing the performance data D into performance periods. Note that each unit data Z corresponding to the performance data D may include the score name of the piece of music performed by the performer Ua. The score name is included in the performance information X.

[0062] Furthermore, the instructor Ub does not speak continuously to the performer Ua, but speaks intermittently at appropriate intervals. For example, the instructor Ub stops speaking during the period when the performer Ua is playing. The unit data Z corresponding to the audio data A is data for managing the period on the time axis during which the instructor Ub's speech represented by the audio data A exists (hereinafter referred to as the "speech period"). That is, the speech period is the period during which the instructor Ub actually speaks. For example, each period adjacent to each other on the time axis separated by an interval longer than a predetermined length is divided into a speech period. That is, an interval shorter than the predetermined length in the instructor Ub's speech is included in one speech period. Note that the predetermined length applied to the division of the speech period (i.e., the threshold value for the interval between speech periods) may be a fixed length set in advance or a variable length set arbitrarily by the user U, for example. Furthermore, if the duration of the speech period is shorter than a predetermined threshold (e.g., 1 second), the information processing unit 512 may delete the data within the speech period from the audio data A as noise data. That is, data within an extremely short speech period in the voice data A may be excluded from being stored in the storage device 52.

[0063] The unit data Z corresponding to the voice data A is generated for each utterance period in the utterance by the instructor Ub. Therefore, the unit data Z is generated for the number of utterance periods in the utterance represented by the voice data A.

[0064] Each unit data Z corresponding to the voice data A includes user identification information Fa, device identification information Fb, and information type Fc, as well as period data T and voice data Qa (Qa1, Qa2, ...). The period data T is a timestamp specifying the speech period. Specifically, the period data T specifies the start time and end time of the speech period, starting from the reference point indicated by the synchronization data. The voice data Qa is the portion of the voice data A within the speech period specified by the period data T. In other words, the voice data Qa is time-series data representing the speech of the instructor Ub within the speech period. As can be understood from the above explanation, each unit data Z includes voice data Qa obtained by dividing the voice data A into speech periods.

[0065] As explained above, the information processing unit 512 generates the period data T that specifies the performance period from the performance data D, and generates the period data T that specifies the speech period from the voice data A.

[0066] The performance evaluation unit 513 in FIG. 5 evaluates the performance of performer Ua represented by performance data D. Specifically, the performance evaluation unit 513 evaluates the skill of the performance by performer Ua by comparing the performance represented by performance data D with an exemplary performance represented by reference data (hereinafter referred to as an "exemplary performance"). The reference data is music data representing the musical score of a piece of music. Specifically, the reference data is, for example, a time series of event data conforming to the MIDI standard. Any known technology may be employed to evaluate the performance by the performance evaluation unit 513.

[0067] The performance evaluation unit 513 identifies a period of the performance represented by the performance data D that is closest to the model performance (hereinafter referred to as the "model period") and generates period data T that specifies the model period. The model period is a period during which the performer Ua was able to perform appropriately. For example, a period during which an evaluation value corresponding to the difference between the performance by the performer Ua and the model performance exceeds a predetermined threshold is identified as the model period. The period data T specifies the start time and end time of the model period, starting from the reference point indicated by the synchronization data. In the following description, the performance period, speech period, and model period are collectively referred to as the "target period."

[0068] Furthermore, the performance evaluation unit 513 identifies periods of high difficulty (hereinafter referred to as "high difficulty periods") in the performance represented by the reference data, and generates period data T that specifies the high difficulty periods. High difficulty periods are, for example, periods in which the number of notes per unit time exceeds a predetermined threshold (for example, periods in which the note density is high due to chords or tuplets, etc.), periods in which the note range is wide, periods requiring a wide range of fingering, periods in which key changes occur, etc.

[0069] The display control unit 514 causes various images to be displayed on the display device 34 of the terminal device 30. Specifically, the display control unit 514 causes the image to be displayed on the display device 34 of the terminal device 30 by transmitting image data representing the image to the terminal device 30. For example, an image instructed by the display control unit 514 is displayed on the terminal device 30 used by the performer Ua or the terminal device 30 used by the instructor Ub.

[0070] The display control unit 514 of the first embodiment displays a reference screen 61 on the display device 34 of the terminal device 30. The reference screen 61 is an image showing the results of analyzing the performance by the performer Ua. FIG. 9 is a schematic diagram of the reference screen 61. The reference screen 61 includes a target person name 611, a musical score image 612, and a plurality of areas G (G1 to G4) corresponding to different types of target periods. The target person name 611 is the name of the performer Ua, "UserA."

[0071] A time axis τ is set on the reference screen 61. The time axis τ represents a period of a series of performances by the performer Ua. Specifically, the time axis τ corresponds to the period of one lesson at a music school, for example. That is, for example, the start point of a lesson at a music school corresponds to the start point of the time axis τ, and the end point of the lesson corresponds to the end point of the time axis τ. Note that the time axis τ may correspond to the login period of the terminal device 30, for example. That is, for example, the time point at which the terminal device 30 logs in corresponds to the start point of the time axis τ, and the time point at which the terminal device 30 logs out corresponds to the end point of the time axis τ.

[0072] An indicator 613 is set on the time axis τ. The indicator 613 is a pointer that specifies a period of the performance by the performer Ua that is to be displayed on the reference screen 61. The score image 612 is a musical score of a predetermined section of the music piece performed by the performer Ua that includes the time point indicated by the indicator 613. The user U can use the operation device 35 to move the indicator 613 along the time axis τ. The score image 612 that corresponds to the position of the indicator 613 on the time axis τ is displayed on the reference screen 61. Therefore, the time axis of the score image 612 has a different scale from the time axis τ of the reference screen 61.

[0073] High difficulty images 614 are displayed on the musical score image 612. The high difficulty images 614 are images that represent high difficulty periods in the musical piece represented by the musical score image 612, and are displayed for each high difficulty period.

[0074] Each of the multiple areas G is a rectangular area that is elongated in the direction of the time axis τ. The multiple areas G are arranged in a direction perpendicular to the time axis τ. The time axis τ is shared among the multiple areas G. The multiple areas G include a performance area G1, an evaluation area G2, a speech area G3, and a video area G4.

[0075] The performance area G1 is an area where the performance period is displayed. Specifically, the performance area G1 displays a display item name N1 and a period image B1. The display item name N1 is the name of the display object, "Performance." The period image B1 is an image representing a performance period in which a performance by the performer Ua exists, and is displayed for each performance period. The position and display length of the period image B1 in the direction of the time axis τ are set according to the start time and end time specified by the period data T.

[0076] The evaluation area G2 is an area where the model period is displayed. Specifically, the evaluation area G2 displays a display item name N2 and a period image B2. The display item name N1 is the name of the display object, "model performance." The period image B2 is an image representing the model period in which the performer Ua was able to perform appropriately, and is displayed for each model period. The position and display length of the period image B1 in the direction of the time axis τ are set according to the start time and end time specified by the period data T between the model periods.

[0077] The speech area G3 is an area where the speech period is displayed. Specifically, the speech area G3 displays a display item name N3 and a period image B3. The display item name N3 is the name of the display object, "Instructor Explanation." The period image B3 is an image representing the speech period in which the instructor Ub spoke, and is displayed for each speech period. The position and display length of the period image B3 in the direction of the time axis τ are set according to the start time and end time specified by the period data T.

[0078] Video area G4 is an area where the video represented by video data V is displayed in chronological order along time axis τ. Display item name N4 is displayed in video area G4. Display item name N4 is the name of the display object, "Video."

[0079] Each period image B (B1 to B3) is displayed in a different display mode. The display mode refers to an image characteristic that can be visually distinguished by the user U. For example, display color, pattern (design), size, or shape are examples of the display mode. The display color is defined by hue (tone), saturation, or brightness (gradation). The display mode of each period image B is changed in response to an instruction from the user U to the operation device 35. For example, the user U can select the display mode of the period image B in each area G from multiple candidates displayed on the display device 34 by selecting a display item name N (N1 to N4) in each area G. Note that the character string or display mode of the display item name N (N1 to N4) of each area G may be changed in response to, for example, an operation on the operation device 35.

[0080] As illustrated above, the reference screen 61 is an image that displays the performance period (G1), the model period (G2), the speech period (G3), and the video of the video data V on a common time axis τ. For example, the performance period (G1) and the speech period (G3) are displayed on a common time axis τ. Therefore, according to the first embodiment, the user U can visually and intuitively grasp the general relationship on the time axis τ between the performance period in which the performer Ua is performing and the speech period in which speech related to the performance is present.

[0081] Furthermore, the display control unit 514 displays the reference screen 61 using the period data T. That is, the period data T that specifies the performance period and the speech period is generated, and the reference screen 61 is displayed using the period data T. The reference screen 61 can also be expressed as an index display related to the performance by the performer Ua.

[0082] In the first embodiment, the model period corresponding to the performance by the performer Ua that is similar to the model performance is displayed on the same time axis τ as the performance period and the speech period, allowing the user U to visually and intuitively grasp the appropriate performance period among the performances by the performer Ua.

[0083] The user U can select a desired target period from among multiple target periods (performance period, speech period, model period) on the reference screen 61 by operating the operation device 35 while viewing the reference screen 61 displayed on the display device 34. For example, the user U can select the target period corresponding to one of the multiple period images B on the reference screen 61 by tapping the specified period image B. The selection receiving unit 515 in FIG. 5 receives from the user U a selection of one of the multiple target periods on the reference screen 61. That is, the selection receiving unit 515 receives a selection of one of the multiple performance periods (G1), one of the multiple model periods (G2), or one of the multiple speech periods (G3).

[0084] When the selection receiving unit 515 receives the selection of a target period, the information output unit 516 outputs detailed information E relating to the target period. For example, when period image B1 or period image B2 is selected by the user U, the information output unit 516 displays, on the display device 34, an image (hereinafter referred to as "performance image E1") representing a portion of the performance data D within the target period represented by the period image B, as detailed information E, as illustrated in FIG. 9. In other words, each period image B on the reference screen 61 functions as an index for the user U to select the target period for which detailed information E should be displayed. FIG. 9 illustrates detailed information E (E1, E2) together with the reference screen 61.

[0085] For example, when the period image B1 is selected, the information output unit 516 displays a performance image E1 representing a portion of the performance data D within the performance period corresponding to the period image B1 on the display device 34 as detailed information E. The performance image E1 is a piano roll image representing the performance data Qd. That is, the performance image E1, in which note images (note bars) representing each note are displayed within an area in which a pitch axis (horizontal axis) and a time axis (vertical axis) are set, is displayed on the display device 34 as detailed information E.

[0086] Similarly, when the period image B2 is selected, the information output unit 516 displays, on the display device 34, a performance image E1 that represents a portion of the performance data D within the model period corresponding to the period image B2, as detailed information E. Note that when the period image B1 or the period image B2 is selected, an image represented by the video data V may be displayed in the video area G4.

[0087] Furthermore, when the period image B3 is selected, the information output unit 516 emits a portion of the audio data A within the speech period corresponding to the period image B3 (hereinafter referred to as "partial audio E2") from the sound emitting device 36. Specifically, the information output unit 516 transmits audio data Qa of the speech period corresponding to the period image B3 selected by the user U to the terminal device 30. Therefore, the partial audio E2 of the instructor Ub within the speech period is reproduced from the sound emitting device 36 as detailed information E.

[0088] As described above, detailed information E corresponding to the period image B selected by the user U from among the plurality of period images B is output (displayed or sounded). Therefore, the user U can check the detailed content of the performance by the performer Ua for the desired period from among the plurality of periods (performance period, model period, speech period, video period).

[0089] Fig. 10 is a flowchart of the process (hereinafter referred to as "control process") executed by the control device 51. Fig. 11 is a schematic diagram of an image displayed on the display device 34 of the terminal device 30 in conjunction with the progress of the control process. For example, the control process is started upon receipt of a login request sent by the terminal device 30. The terminal device 30 can establish communication with the control system 50 by logging in to the control system 50.

[0090] To log in to the control system 50, the control device 51 displays an authentication screen 621 on the display device 34 of the terminal device 30 (Sb1). The authentication screen 621 is a screen for inputting, for example, authentication information of the user U. The authentication information includes, for example, identification information and a password. Note that the user U is only required to input the authentication information the first time he logs in, and input of the authentication information may be omitted from the second and subsequent times he logs in.

[0091] The authentication process may be biometric authentication using biometric information of the user U. The biometric information is, for example, a facial image (facial authentication) or a voice signal (voice authentication) of the user U.

[0092] Facial recognition uses a recognition technology that utilizes a statistical model such as a deep neural network or HMM. For example, facial recognition is performed using Haar-like features extracted from a facial image. Note that general-purpose software may also be used for facial recognition.

[0093] Voice authentication utilizes a recognition technology that utilizes a statistical model, such as a deep neural network or HMM. For example, after performing voice recognition using the statistical model, voice authentication is performed using voice features for each individual. Note that voice authentication may also be performed using general-purpose software.

[0094] In addition, a similar login from the terminal device 30 to the control system 50 is required not only when the terminal device 30 starts displaying the reference screen 61, but also when various data related to the performance by the performer Ua is sent to the control system 50.

[0095] The control device 51 displays a device setting screen 622 on the display device 34 of the terminal device 30 (Sb2). The device setting screen 622 is an image that allows the user U to set each of a plurality of types of peripheral devices to be enabled or disabled. The user U can select whether to enable or disable each peripheral device by operating the operation device 35. Data of peripheral devices that are set to be enabled on the device setting screen 622 are to be displayed on the reference screen 61.

[0096] As illustrated in FIG. 11, the control device 51 displays a saved data list 623 on the display device 34 of the terminal device 30 (Sb3). The saved data list 623 is a list of recorded data R1 previously saved by the user U. Specifically, the save date and time and duration are displayed for each previously saved data R1. The user U can select one of the multiple recorded data R1 displayed in the saved data list 623 by operating the operation device 35. The control device 51 (display control unit 514) selects management data R3 corresponding to the recorded data R1 selected by the user U from the storage device 52 (Sb4). The control device 51 (display control unit 514) uses the management data R3 selected from the storage device 52 to display a reference screen 61 on the display device 34 of the terminal device 30 (Sb5).

[0097] The control device 51 (selection receiving unit 515) receives a selection of one of a plurality of target periods from the user U (Sb6). The control device 51 (information output unit 516) outputs detailed information E corresponding to the target period selected by the user U via the terminal device 30 (Sb7). The operation of the control system 50 in the first embodiment is as described above.

[0098] B: Second embodiment A second embodiment will be described. Note that, for elements in the following exemplary aspects that have the same functions as those in the first embodiment, the same reference numerals as those in the first embodiment will be used, and detailed descriptions of each will be omitted as appropriate.

[0099] 12 is a schematic diagram of a reference screen 61 and detailed information E in the second embodiment. The reference screen 61 in the second embodiment is similar to that in the first embodiment. The detailed information E in the second embodiment includes a plurality of pieces of information with different information types Fc.

[0100] When the selection receiving unit 515 receives a selection of one of multiple target periods (hereinafter referred to as the "selected period") from the user U, the information output unit 516 selects one or more other target periods that overlap at least a portion of the selected period on the time axis τ. Specifically, the selection of one selected period by the user U triggers the selection of multiple target periods that include the selected period. In other words, the information output unit 516 groups multiple target periods that include the selected period selected by the user U and target periods that overlap with the selected period.

[0101] The information output unit 516 displays detailed information E corresponding to each of the multiple target periods in parallel on the display device 34. FIG. 12 illustrates a situation in which the user U selects one performance period as the selected period. In the above situation, in addition to the performance image E1 of the performance period selected by the user U, a partial video E3 and a performance parameter transition E4 are displayed on the display device 34 as detailed information E. The partial video E3 is a partial video within the target period that overlaps with the performance period among the videos represented by the video data V. Specifically, multiple moving images captured from different directions of the performance of the keyboard instrument 10 by the performer Ua are displayed as the partial video E3.

[0102] The performance parameter transition E4 is an image showing the change in the performance parameter over time. The performance parameter is, for example, a parameter related to the performance represented by the performance data D. For example, the strength (velocity) of the key depression by the performer Ua is an example of the performance parameter. Note that the multiple pieces of detailed information E may be displayed in chronological order of the time specified by the period data T.

[0103] The second embodiment also achieves the same effects as the first embodiment. Furthermore, in the second embodiment, detailed information E relating to multiple target periods is output by selecting one target period. Therefore, compared to a configuration (for example, the first embodiment) in which a target period must be selected individually for each desired detailed information E, the operational burden on the user U can be reduced.

[0104] C: Third embodiment 13 is a schematic diagram of a reference screen 61 and detailed information E in the third embodiment. The reference screen 61 in the third embodiment is similar to that in the first embodiment. When a selection of an utterance period is received from the user U, the information output unit 516 in the third embodiment displays detailed information E representing the content of the utterance of the instructor Ub during the utterance period on the display device 34. Specifically, the information output unit 516 generates an instruction string E5 representing the content of the instruction to the performer Ua.

[0105] The information output unit 516 first identifies a spoken character string by performing voice recognition on the voice data Qa corresponding to the speech period. The spoken character string is a character string that represents the content of the speech made by the instructor Ub. Any known technology may be used for voice recognition on the voice data Qa. The information output unit 516 extracts a character string containing a predetermined specific phrase (hereinafter referred to as a "specific phrase") from the spoken character string. For example, a character string containing a specific phrase such as "today's point" or "task" is extracted from the spoken character string.

[0106] The information output unit 516 acquires a standard string stored in advance in the storage device 52. The standard string is a standard string representing instruction to the performer Ua. The standard string is a string in which the portion corresponding to the specific phrase is left blank. The information output unit 516 generates an instruction string E5 by inserting a character string extracted from the spoken string into the blank corresponding to the specific phrase in the standard string. Therefore, the instruction string E5 is a string summarizing the portion of the speech uttered by the instructor Ub, particularly the portion related to the specific phrase. The instruction string E5 is displayed on the display device 34 as detailed information E. The instruction string E5 is used by the performer Ua as lesson notes related to the instruction by the instructor Ub. As described above, the instruction string E5 is composed of a combination of a standard string prepared in advance and a portion of the speech string spoken by the instructor Ub. Note that the instruction string E5 may be edited in response to an instruction from the user U via the operation device 35.

[0107] The information output unit 516 also generates an edited video E6 by combining (for example, connecting so as to be temporally continuous) portions of the video represented by the video data V and the audio represented by the audio data A within each target period selected by the user U. The edited video E6 is a digest video made up of video and audio representing the performance by the performer Ua. Note that the information output unit 516 may include, in the edited video E6, a section of the video represented by the video data V in which the instructor Ub utters a specific phrase. The information output unit 516 displays the edited video E6 on the display device 34 of the terminal device 30.

[0108] The third embodiment also achieves the same effects as the first embodiment. Furthermore, in the third embodiment, an instruction string E5 representing instruction by instructor Ub is displayed on the display device 34, so the user U can visually confirm the instruction by instructor Ub. Furthermore, an edited video E6 is displayed on the display device 34, so the user U (performer Ua or instructor Ub) can visually confirm the performance by performer Ua.

[0109] D: Fourth embodiment FIG. 14 is a schematic diagram of an image displayed on the display device 34 of the terminal device 30 that has logged in to the control system 50. When the user U operates the operation device 35 while the saved data list 623 is displayed on the display device 34 (Sb3), the control device 51 (display control unit 514) displays a viewing permission screen 624 on the display device 34. The viewing permission screen 624 is a list of multiple candidates associated with the user U. For example, multiple candidates related to the character string entered by the user U are displayed. The user U operates the operation device 35 to select one or more candidates for whom the reference screen 61 is permitted to be displayed. In other words, candidates not selected by the user U are prohibited from viewing the reference screen 61.

[0110] When a candidate is selected, the control device 51 (display control unit 514) stores the identification information of the user U who is permitted to view the reference screen 61 in the storage device 52. The control device 51 (display control unit 514) displays a saved data list 623′ including the identification information of the user U who is permitted to view the reference screen 61 on the display device 34.

[0111] When a request to display the reference screen 61 is made by the terminal device 30 of the user U, the control device 51 (display control unit 514) determines whether or not the user U is permitted to view the reference screen 61. If viewing of the reference screen 61 is permitted, the control device 51 (display control unit 514) displays the reference screen 61 on the display device 34 of the terminal device 30, as in the above-described embodiments. On the other hand, if viewing of the reference screen 61 is prohibited, the control device 51 (display control unit 514) displays a message on the display device 34 indicating that viewing is prohibited. In other words, the reference screen 61 is not displayed on the display device 34.

[0112] The fourth embodiment also achieves the same effects as the first embodiment. Furthermore, in the fourth embodiment, permission to view the reference screen 61 is set for each user U, so the range of disclosure of each reference screen 61 can be limited.

[0113] E: Fifth embodiment 15 is a schematic diagram of an image displayed on the display device 34 of the terminal device 30 that has logged in to the control system 50. When the user U operates the operation device 35 while the stored data list 623 is displayed on the display device 34 (Sb3), the control device 51 (display control unit 514) displays a period setting screen 625 on the display device 34. The period setting screen 625 is a screen for setting a storage period for each piece of recorded data R1. The storage period is the period for which the recorded data R1 is validly stored in the storage device 52.

[0114] The user U operates the operation device 35 to specify the storage period for each piece of recorded data R1. For example, the storage period is specified by the length of time from the time of setting (e.g., one month) or the date and time of the end of the period. The control device 51 (information processing unit 512) stores the storage period specified by the user U in the storage device 52 in association with the recorded data R1. In addition to setting the storage period, the user U can also prohibit the deletion of the recorded data R1.

[0115] The control device 51 (display control unit 514) determines whether the current date and time is included in the storage period of each recorded data R1. The control device 51 deletes recorded data R1 of the multiple recorded data R1 whose storage period has elapsed. Note that recorded data R1 that is prohibited from being deleted is not deleted.

[0116] The fifth embodiment also achieves the same effects as the first embodiment. Furthermore, in the fifth embodiment, a storage period for the recorded data R1 is set, which reduces the possibility that a large amount of recorded data R1 will remain unused in the storage device 52 for a long period of time. Therefore, the storage capacity required for the storage device 52 can be reduced.

[0117] F: Sixth embodiment 16 is a schematic diagram of a reference screen 61 in the sixth embodiment. The reference screen 61 in the sixth embodiment displays periods in which the performer Ua made performance errors (hereinafter referred to as "performance error periods"). The performance evaluation unit 513 in the sixth embodiment identifies performance error periods by comparing the performance represented by the performance data D with the model performance represented by the reference data. For example, a period in which the number of differences between the performance by the performer Ua and the model performance represented by the reference data exceeds a predetermined threshold is identified as a performance error period.

[0118] The display control unit 514 displays a period image B5 representing the performance error period in the performance area G1 together with the period image B1. Specifically, the performance error period is displayed on a time axis τ common to the performance period and other target periods such as the speech period. Note that while FIG. 16 illustrates an example in which the period image B5 of the performance error period is displayed in the performance area G1, the period image B5 may also be displayed in an area G separate from the performance area G1.

[0119] The sixth embodiment also achieves the same effects as the first embodiment. Furthermore, in the sixth embodiment, periods of performance errors corresponding to performances by performer Ua that differ from the model performance are displayed on a time axis τ common to the performance periods and speech periods. Therefore, the user U can visually and intuitively grasp the periods in the performance by performer Ua in which performance errors occurred.

[0120] G: Seventh embodiment In each of the above-described embodiments, a situation has been assumed for convenience in which a performer Ua plays the keyboard instrument 10 and an instructor Ub speaks to provide instruction. However, in a music classroom, for example, a situation in which an instructor Ub plays the keyboard instrument 10 in addition to the performer Ua, or a situation in which the performer Ua speaks in addition to the instructor Ub, can also be assumed. The seventh embodiment is a form for displaying a reference screen 61 related to the performances and speeches by both the performer Ua and the instructor Ub.

[0121] 17 is a schematic diagram of an image displayed on the display device 34 of the terminal device 30 that has logged in to the control system 50. An authentication screen 671 is displayed on the display device 34 of the terminal device 30, allowing the instructor Ub to input authentication information (e.g., identification information and a password). If the authentication process by the instructor Ub is successful, an answer screen 672 is displayed, prompting the instructor Ub to input whether there are other users U with whom the recorded data R1 is to be shared. If the instructor Ub answers that the recorded data R1 is to be shared with other users U, an authentication screen 673 is displayed, allowing the performer Ua to input authentication information. By repeating the display transitions illustrated above, each of the multiple users U (performers Ua, instructors Ub) can log in.

[0122] As illustrated in FIG. 18, after the instructor Ub authentication screen 671 is displayed, an instruction screen 674 for designating one or more performers Ua who will share the recording data R1 with the instructor Ub, and a confirmation screen 675 for confirming whether the designation of the performers Ua is appropriate may be displayed.

[0123] The performance of the keyboard instrument 10 represented by the performance data D of the seventh embodiment includes a performance by the performer Ua and a performance by the instructor Ub. The speech represented by the voice data A includes a speech by the performer Ua and a speech by the instructor Ub.

[0124] The method for distinguishing between the performance by the performer Ua and the performance by the instructor Ub is arbitrary. For example, either the performer Ua or the instructor Ub may be designated by operating the operation device 15 of the keyboard instrument 10 or the operation device 35 of the terminal device 30. The control device 51 (information processing unit 512) distinguishes between the performance by the performer Ua and the performance by the instructor Ub in accordance with the operation of the operation device 15 or the operation device 35.

[0125] Furthermore, any method may be used to distinguish between the speech of the performer Ua and the speech of the instructor Ub. For example, either the performer Ua or the instructor Ub may be designated by operating the operation device 15 of the keyboard instrument 10 or the operation device 35 of the terminal device 30. The control device 51 (information processing unit 512) distinguishes between the speech of the performer Ua and the speech of the instructor Ub in accordance with the operation of the operation device 15 or the operation device 35. Note that the speech of the performer Ua and the speech of the instructor Ub may also be distinguished by speaker identification of the audio data A.

[0126] 19 is a schematic diagram of a reference screen 61 in the seventh embodiment. The reference screen 61 in the seventh embodiment includes a performance area G1a, a performance area G1b, a speech area G3a, and a speech area G3b. Note that, for convenience, other areas G such as an evaluation area G2 and a video area G4 are omitted from FIG. 19, but may be included in the reference screen 61 as in the previous embodiments.

[0127] The performance area G1a is an area where the performance period related to the performance of the performer Ua is displayed. Specifically, the performance area G1a displays a display item name N1 and a period image B1. The display item name N1 includes the name of the performer Ua, "UserA," and the name of the display object, "Performance." The period image B1 is an image that represents the performance period in which the performance by the performer Ua exists.

[0128] The performance area G1b is an area where the performance period related to the performance of instructor Ub is displayed. Specifically, the performance area G1b displays a display item name N1 and a period image B1. The display item name N1 includes the name of instructor Ub, "UserB," and the name of the display object, "Performance." The period image B1 is an image that represents the performance period in which instructor Ub is performing.

[0129] As explained above, the multiple performance periods displayed on the reference screen 61 of the seventh embodiment include a performance period in which a performance by performer Ua is included and a performance period in which a performance by instructor Ub is included. Note that a performance by performer Ua is an example of a "first performance," and a performance period by performer Ua is an example of a "first performance period." Also, a performance by instructor Ub is an example of a "second performance," and a performance period by instructor Ub is an example of a "second performance period."

[0130] In the seventh embodiment, a performance period in which a performance by performer Ua exists and a performance period in which a performance by instructor Ub exists are displayed on a common time axis τ on the display device 34. Therefore, the user U can visually and intuitively grasp the temporal relationship between the performance by performer Ua and the performance by instructor Ub.

[0131] The speech area G3a is an area where a speech period related to speech by the performer Ua is displayed. Specifically, the speech area G3a displays a display item name N3 and a period image B3. The display item name N3 includes the name of the performer Ua, "UserA," and the name of the display object, "Voice." The period image B3 is an image that represents a speech period in which speech by the performer Ua exists.

[0132] The speech area G3b is an area where a speech period related to the speech of instructor Ub is displayed. Specifically, the speech area G3b displays a display item name N3 and a period image B3. The display item name N3 includes the name of instructor Ub, "UserB," and the name of the display object, "Voice." The period image B3 is an image that represents a speech period in which instructor Ub speaks.

[0133] As explained above, the multiple speech periods displayed on the reference screen 61 of the seventh embodiment include a speech period in which the speech voice of the performer Ua is present and a speech period in which the speech voice of the instructor Ub is present. Note that the speech voice of the performer Ua is an example of a "first speech voice," and the speech period of the performer Ua is an example of a "first speech period." Also, the speech voice of the instructor Ub is an example of a "second speech voice," and the speech period of the instructor Ub is an example of a "second speech period."

[0134] In the seventh embodiment, the speech period in which the speech voice of the performer Ua exists and the speech period in which the speech voice of the instructor Ub exists are displayed on a common time axis τ on the display device 34. Therefore, the user U can visually and intuitively grasp the temporal relationship between the speech voice of the performer Ua and the speech voice of the instructor Ub.

[0135] For example, in a music school, one instructor Ub may instruct multiple performers Ua. The performances and speeches of multiple performers Ua (Ua1, Ua2) who share the same instructor Ub may be displayed on the reference screen 61. For example, a performance period in which a performance by performer Ua1 is present and a performance period in which a performance by performer Ua2 is present are displayed on the display device 34 along a common time axis τ. Also, for example, a speech period in which a speech voice of performer Ua1 is present and a speech period in which a speech voice of performer Ua is present are displayed on the display device 34 along a common time axis τ. Also, the performances and speeches of multiple performers Ua and instructor Ub may be displayed in parallel on the reference screen 61.

[0136] It is also possible for the information processing system 100 to include multiple sets of keyboard instruments 10 and terminal devices 30. The above-described embodiments are also applicable to situations in which multiple performers Ua or multiple instructors Ub each use the keyboard instrument 10 and the terminal device 30.

[0137] H: Eighth embodiment As described above, the recorded data R1 for each of the multiple users U (musicians Ua and instructors Ub) is stored in the storage device 52. The eighth embodiment is a form for comparing the performances of the multiple users U.

[0138] 20 is a schematic diagram of an image displayed on the display device 34 of the terminal device 30 in the eighth embodiment. The control device 51 (display control unit 514) displays a selection screen 681 on the display device 34 in response to an operation from the user U on the operation device 35. The selection screen 681 is an image for selecting multiple users U to be compared (hereinafter referred to as "display target persons"). For example, a performer Ua or an instructor Ub is selected as the display target person.

[0139] When multiple display targets are selected, the control device 51 (display control unit 514) contrastively displays detailed information E for each of the multiple display targets. In Fig. 20, a performance image E1, a partial video E3, and a performance parameter transition E4 are displayed on the display device 34 for both the performer Ua and the instructor Ub.

[0140] In FIG. 20, performer Ua and instructor Ub are assumed to be the display subjects. A performance image E1 of instructor Ub is superimposed on a performance image E1 of performer Ua. Therefore, the user U can easily compare the performance of the keyboard instrument 10 between performer Ua and instructor Ub. Furthermore, a performance parameter transition E4 of instructor Ub is superimposed on a performance parameter transition E4 of performer Ua. Therefore, the user U can easily compare the performance parameter transition E4 between performer Ua and instructor Ub.

[0141] The eighth embodiment also achieves the same effects as the first embodiment. Furthermore, in the eighth embodiment, detailed information E is displayed comparatively for multiple display subjects, so that the user U can visually compare the performances of the display subjects.

[0142] I: Ninth embodiment 21 is a schematic diagram of an image displayed on the display device 34 of the terminal device 30 in the ninth embodiment. The control device 51 (display control unit 514) displays a selection screen 691 on the display device 34 in response to an operation from the user U on the operation device 35. The selection screen 691 is an image for selecting multiple display target persons, similar to the eighth embodiment.

[0143] When multiple display targets are selected, the control device 51 (display control unit 514) displays a ratio image 692 on the display device 34. The ratio image 692 is a pie chart that displays, for each of the multiple display targets, the time ratio of the performance period and the speaking period relative to the duration of one lesson in the music studio. Specifically, the ratio image 692, which represents the time ratio of the performance period of the performer Ua (UserA / performance), the time ratio of the speaking period of the performer Ua (UserA / voice), the time ratio of the performance period of the instructor Ub (UserB / performance), and the time ratio of the speaking period of the instructor Ub (UserA / voice), is displayed on the display device 34. By referring to the ratio image 692, the user U can visually and intuitively grasp the structure of one lesson.

[0144] Furthermore, the control device 51 (display control unit 514) displays the selection screen 693 of Fig. 22 on the display device 34 in response to an operation by the user U on the operation device 35. As in the eighth embodiment, the selection screen 693 is an image for selecting one display target person from a plurality of candidates.

[0145] When the display target person is selected, the control device 51 (display control unit 514) displays a ratio image 694 on the display device 34. The ratio image 694 is a pie chart that displays the time ratio between the performance period and the speech period of the display target person. For example, the time ratio of the performance period of instructor Ub (UserB / performance) and the time ratio of the speech period of instructor Ub (UserB / voice) are displayed on the display device 34. By referring to the ratio image 694, the user U can visually and intuitively grasp the temporal relationship between the performance period and the speech period of the display target person. Therefore, the ratio image 694 is used as an effective reference material for evaluating the appropriateness of instruction by instructor Ub, who is the display target person, for example.

[0146] As can be understood from the above example, the display control unit 514 of the ninth embodiment displays on the display device 34 ratio images (692, 694) that represent time ratios regarding a plurality of target periods including performance periods and speech periods.

[0147] J: Variation Specific modified embodiments that can be added to each of the embodiments exemplified above are exemplified below. Two or more embodiments arbitrarily selected from the following examples may be combined as appropriate within the scope of not being mutually contradictory.

[0148] (1) In the above-described embodiments, the performer Ua plays the keyboard instrument 10, but the instrument played by the performer Ua is not limited to the keyboard instrument 10. In other words, the present disclosure can be applied to managing the performance of any type of instrument.

[0149] (2) In the above-described embodiments, multiple moving images captured from different directions of the performer Ua playing the keyboard instrument 10 are used as an example of the partial video E3 ( FIG. 12 ). However, the content of the partial video E3 is not limited to the above examples. For example, partial video E3 may include a video of both hands of the performer Ua playing the keyboard instrument 10, a video including the performer Ua and the instructor Ub, or a video of the performer Ua's feet operating the pedals 184. Similarly, the edited video E6 is not limited to a video of both hands of the performer Ua.

[0150] (3) In the above-described embodiments, piano roll images are used as examples of the performance image E1 and the performance parameter transition E4, but the specific embodiments of the performance image E1 and the performance parameter transition E4 are not limited to these examples. For example, the operation of the pedal 184 by the performer Ua may be displayed by the performance image E1 or the performance parameter transition E4. For example, a performance image E1 representing the period during which the performer Ua operates the pedal 184 may be displayed. Furthermore, in an embodiment in which the performance parameter includes the amount of depression of the pedal 184, the amount of depression of the pedal 184 by the performer Ua may be displayed as the performance parameter transition E4.

[0151] (4) In each of the above-described embodiments, the reference data representing the model performance is, for example, data prepared in advance. However, performance data D of the instructor Ub who performs together with the performer Ua may also be used as the reference data. For example, when the performer Ua and the instructor Ub log in to the control system 50 using the terminal device 30, an operating mode can be set in which performance data D output from the keyboard instrument 10 of the instructor Ub is used as reference data. The control device 51 (recorded data acquisition unit 511) stores the performance data D representing the performance of the instructor Ub in the storage device 52 as new reference data.

[0152] In a situation where the performer Ua and the instructor Ub are playing separate or parallel keyboard instruments 10, the recorded data acquisition unit 511 may store performance data D generated by the performance of the instructor Ub in the storage device 52 as reference data, and the performance evaluation unit 513 may evaluate the performance of the performer Ua in real time using the reference data. In other words, the acquisition of the reference data and the evaluation of the performance of the performer Ua may be performed in real time in parallel with the performances by the performer Ua and the instructor Ub.

[0153] (5) In each of the above-described embodiments, the information processing system 100 includes a keyboard instrument 10, but the keyboard instrument 10 may be omitted from the information processing system 100. For example, the keyboard instrument 10 may be installed in an acoustic space located remotely from the information processing system 100 (terminal device 30). The performance data D generated by the keyboard instrument 10 is transmitted to the information processing system 100 via a communication network 200 such as the Internet. The operation of the terminal device 30 is the same as in each of the above-described embodiments. As can be understood from the above explanation, the "information processing system" in this disclosure may or may not include a musical instrument.

[0154] (6) Some or all of the functions of the control system 50 illustrated in each of the above embodiments may be installed in the terminal device 30.

[0155] (7) As mentioned above, the functions of the control system 50 illustrated above are realized by cooperation between one or more processors constituting the control device 51 and a program stored in the storage device 52. The program according to the present disclosure may be provided in a form stored on a computer-readable recording medium and installed on a computer. The recording medium may be, for example, a non-transitory recording medium, such as an optical recording medium (optical disk) such as a CD-ROM, but may also include any known type of recording medium, such as a semiconductor recording medium or a magnetic recording medium. Note that a non-transitory recording medium includes any recording medium other than a transitory, propagating signal, and does not exclude volatile recording media. Furthermore, in a configuration in which a distribution device distributes a program via the communication network 200, the storage medium that stores the program in the distribution device corresponds to the non-transitory recording medium described above.

[0156] K:Additional notes From the above-described exemplary embodiments, the following configurations can be understood, for example.

[0157] An information processing system according to one aspect (aspect 1) of the present disclosure includes a recorded data acquisition unit that acquires performance data representing a musical instrument performance by a musician and audio data representing speech related to the performance by the musician, and a display control unit that displays on a display device a reference screen that represents, on a common time axis, one or more performance periods in which the performance represented by the performance data exists and one or more speech periods in which the speech represented by the audio data exists. In the above aspect, a user (e.g., a musician or instructor) can visually and intuitively grasp the general relationship on the time axis between the performance periods in which the musician performs and the speech periods in which the speech related to the performance exists.

[0158] "Performance data" is data in any format that represents a performance of an instrument by a performer. For example, MIDI time-series data representing each note played by a performer, or audio data recording the sound of a performer playing an instrument, are examples of "performance data."

[0159] "Audio data" is data in any format that represents the voice (spoken voice) uttered by a performer at the location where the performer is performing. For example, data representing the voice of the performer himself or the voice of an instructor who is instructing the performer are examples of "audio data." Furthermore, data representing the voice of a listener listening to the performance of a performer is also included in "audio data."

[0160] The recorded data acquisition unit can acquire performance data representing a performance by a single performer, as well as multiple performance data representing performances of musical instruments by different performers. The source from which the recorded data acquisition unit acquires the performance data is arbitrary. For example, the recorded data acquisition unit may acquire the performance data from a device directly connected to the information processing system, or from a remote device that communicates with the information processing system.

[0161] The recorded data acquisition unit can acquire audio data representing speech from a single location, as well as multiple audio data representing speech from different locations. The source from which the recorded data acquisition unit acquires audio data is arbitrary. For example, the recorded data acquisition unit may acquire audio data from a device directly connected to the information processing system, or from a remote device that communicates with the information processing system.

[0162] A "performance period" is a continuous period during which a musician plays an instrument, while a "speech period" is a continuous period during which speech sounds are produced. A "speech period" may include, for example, short periods of silence between speech utterances.

[0163] The "reference screen" is an image of any format that represents a performance period and a speech period on a time axis. For example, an image that displays a performance period and a speech period on a common time axis is an example of the "reference screen."

[0164] In a specific example (Aspect 2) of Aspect 1, an information processing unit is further provided that generates period data specifying the performance period and the utterance period using the performance data and the voice data, and the display control unit displays the reference screen using the period data. In the above aspect, the period data specifying the performance period and the utterance period is generated, and the reference screen is displayed using the period data.

[0165] The "period data" is data that specifies the start and end times of each period on the time axis. For example, data that specifies the start and end times of each performance period and speech period is exemplified as the "period data."

[0166] In a specific example (Aspect 3) of Aspect 1 or Aspect 2, the one or more performance periods are a plurality of performance periods, and the device further includes a selection receiving unit that receives a selection of one of the plurality of performance periods on the reference screen, and an information output unit that outputs detailed information indicating the content of the performance within the performance period when the selection receiving unit receives the selection of the performance period. In the above aspect, the detailed content of the performance can be confirmed for a performance period desired by the user out of the plurality of performance periods.

[0167] The "detailed information" is an image or sound in any format that represents the content of the performance during the performance period. For example, the "detailed information" may be a performance image (e.g., a piano roll image) that represents a string of musical notes during the performance period, a performance video that captures the performance of the performer during the performance period, or sound that represents these images.

[0168] In a specific example (Aspect 4) of any one of Aspects 1 to 3, the speech represented by the audio data includes a first speech of the performer and a second speech of the instructor instructing the performer, and the one or more speech periods on the reference screen include a first speech period in which the first speech exists and a second speech period in which the second speech exists. In the above aspect, the first speech period in which the first speech of the performer exists and the second speech period in which the second speech of the instructor instructing the performer exists are displayed. Therefore, the temporal relationship between the first speech and the second speech can be visually and intuitively grasped.

[0169] In a specific example (Aspect 5) of any of Aspects 1 to 4, the performance represented by the performance data includes a first performance by a first performer and a second performance by a second performer different from the first performer, and the one or more performance periods on the reference screen include a first performance period in which the first performance occurs and a second performance period in which the second performance occurs. In the above aspects, the first performance period in which the first performance by the first performer occurs and the second performance period in which the second performance by the second performer occurs are displayed. Therefore, the temporal relationship between the first performance and the second performance can be visually and intuitively grasped.

[0170] The first performer and the second performer include a combination of two different performers, as well as a combination of a performer and an instructor. The first performance by the first performer and the second performance by the second performer may be performances performed in parallel with each other, or may be performances recorded sequentially at different times.

[0171] In a specific example (Aspect 6) of any of Aspects 1 to 5, a performance evaluation unit is further provided that compares the performance represented by the performance data with the model performance represented by the reference data, and the reference screen displays one or more model periods corresponding to performances represented by the performance data that are similar to the model performance on a common time axis with the one or more performance periods and the one or more speech periods. In the above aspect, the model periods corresponding to performances by the performer that are similar to the model performance are displayed on a common time axis with the performance periods and the speech periods. This allows the user to visually and intuitively grasp the appropriate performance periods in the performer's performance.

[0172] In a specific example (Aspect 7) of any of Aspects 1 to 5, a performance evaluation unit is further provided that compares the performance represented by the performance data with the model performance represented by the reference data, and the reference screen displays one or more periods of performance error corresponding to the performance represented by the performance data that differs from the model performance on a time axis common to the one or more performance periods and the one or more speech periods. In the above aspect, the periods of performance error corresponding to the performance by the performer that differs from the model performance are displayed on a time axis common to the performance periods and the speech periods. Therefore, the user can visually and intuitively grasp the periods in the performer's performance where performance errors occurred.

[0173] In a specific example (Aspect 8) of any of Aspects 1 to 7, the display control unit further displays, on the display device, a ratio image representing the time ratios of multiple periods including the one or more performance periods and the one or more speech periods. In the above aspects, a user can visually and intuitively grasp the time ratios of each period including one or more performance periods and one or more speech periods. The time ratios of each period are useful, for example, for determining whether a performer's performance practice is appropriate or whether an instructor's performance instruction is appropriate.

[0174] An information processing method according to one aspect (aspect 9) of the present disclosure acquires performance data representing a performance of an instrument by a performer and audio data representing speech related to the performance by the performer, and displays on a display device a reference screen showing, on a common time axis, one or more performance periods in which the performance represented by the performance data exists and one or more speech periods in which the speech represented by the audio data exists.

[0175] A program according to one aspect (aspect 10) of the present disclosure causes a computer system to function as a recorded data acquisition unit that acquires performance data representing a performance of an instrument by a performer and audio data representing speech related to the performance by the performer, and a display control unit that displays on a display device a reference screen that represents, on a common time axis, one or more performance periods in which the performance represented by the performance data exists and one or more speech periods in which the speech represented by the audio data exists. [Explanation of symbols]

[0176] 100...information processing system, 200...communication network, 10...keyboard instrument, 11...control device, 12...storage device, 13...communication device, 14...display device, 15...operation device, 16...sound source device, 17...sound emission device, 18...performance mechanism, 181...keyboard, 182...key, 183...sound generation mechanism, 184...pedal, 185...detection device, 20...recording system, 21...imaging device, 22...sound collection device, 30(30,30b,30c)...terminal device, 3 1...control device, 32...storage device, 33...communication device, 34...display device, 35...operation device, 36...sound emission device, 37...sound collection device, 40...measurement system, 41...biometric sensor, 42...center of gravity sensor, 50...control system, 51...control device, 52...storage device, 53...communication device, 511...recorded data acquisition unit, 512...information processing unit, 513...performance evaluation unit, 514...display control unit, 515...selection reception unit, 516...information output unit.

Claims

1. a recorded data acquisition unit that acquires performance data representing a performance of an instrument by a performer and voice data representing a speech related to the performance by the performer; a display control unit that displays, on a display device, a reference screen that represents, on a common time axis, one or more performance periods in which a performance represented by the performance data exists and one or more speech periods in which a speech voice represented by the voice data exists; An information processing system comprising:

2. an information processing unit that generates period data that designates the performance period and the utterance period using the performance data and the voice data; The display control unit displays the reference screen using the period data. The information processing system of claim 1.

3. the one or more performance periods are a plurality of performance periods; a selection receiving unit that receives a selection of one of the plurality of performance periods on the reference screen; an information output unit that outputs detailed information indicating the content of the performance within the performance period when the selection receiving unit receives the selection of the performance period; The information processing system of claim 1 further comprising:

4. the speech represented by the voice data includes a first speech of the player and a second speech of an instructor who is instructing the player, The one or more speech periods in the reference screen include a first speech period in which the first speech voice is present and a second speech period in which the second speech voice is present. The information processing system of claim 1.

5. the performance represented by the performance data includes a first performance by a first performer and a second performance by a second performer different from the first performer; The one or more performance periods on the reference screen include a first performance period in which the first performance exists and a second performance period in which the second performance exists. The information processing system of claim 1.

6. a performance evaluation unit that compares the performance represented by the performance data with a model performance represented by reference data; The reference screen displays one or more model periods corresponding to performances that are similar to the model performance among the performances represented by the performance data, on a time axis common to the one or more performance periods and the one or more utterance periods. The information processing system of claim 1.

7. a performance evaluation unit that compares the performance represented by the performance data with a model performance represented by reference data; The reference screen displays one or more performance error periods corresponding to performances that differ from the model performance among the performances represented by the performance data, on a common time axis with the one or more performance periods and the one or more utterance periods. The information processing system of claim 1.

8. The display control unit further displays, on the display device, a ratio image representing a time ratio regarding a plurality of periods including the one or more performance periods and the one or more speech periods. The information processing system of claim 1.

9. Acquire performance data representing a performance of an instrument by a performer and voice data representing a speech related to the performance by the performer; A reference screen is displayed on a display device, which displays, on a common time axis, one or more performance periods in which the performance represented by the performance data exists and one or more speech periods in which the speech represented by the voice data exists. An information processing method implemented by a computer system.

10. a recorded data acquisition unit that acquires performance data representing a performance of an instrument by a performer and voice data representing a speech related to the performance by the performer; a display control unit that displays, on a display device, a reference screen that represents, on a common time axis, one or more performance periods in which a performance represented by the performance data exists and one or more speech periods in which a speech voice represented by the voice data exists; A program that makes a computer system function as a

Images