Information processing system, information processing method, and program

Abstract

An information processing system has: a collection data acquisition section (511) that acquires performance data indicating performance of a musical instrument by a performer and audio data indicating speech audio related to the performance by the performer; and a display control section (514) that displays a reference screen on a display device, the reference screen displaying, based on a common time axis, one or more performance periods in which there is performance indicated by the performance data and one or more speech periods in which there is speech audio indicated by the audio data.

Description

Technical Field

[0001] This invention relates to a technique to assist performers in practicing musical instruments. Background Technology

[0002] Various techniques for assisting in the practice of musical instruments have been proposed in the past. For example, Patent Document 1 discloses a technique that assists in playing a piano by feeding back information on the coordination of multiple motion elements in the performance to the user.

[0003] Patent Document 1: International Publication No. 2019 / 130755 Summary of the Invention

[0004] However, in scenarios such as when a performer is practicing an instrument, if the time relationship between, for example, the period during which the performer is actually playing the instrument and the period during which the performer is conversing with, for example, an instructor, can be grasped, further efficiency in the instrument practice can be expected. Considering the above, one objective of this invention is to enable the user to visually and intuitively grasp the approximate time relationship between the performance period and the speaking period.

[0005] To address the above-mentioned issues, one aspect of the present invention relates to an information processing system comprising: a data acquisition unit that acquires performance data representing a performer's performance of an instrument and audio data representing speech audio related to the performer's performance; and a display control unit that displays a reference screen on a display device, the reference screen displaying, based on a common timeline, one or more performance periods containing the performance data and one or more speech periods containing the speech audio represented by the audio data.

[0006] One aspect of the present invention relates to an information processing method that obtains performance data representing a performer's performance on an instrument and audio data representing speech audio related to the performer's performance, and displays a reference screen on a display device. The reference screen is based on a common timeline and displays one or more performance periods in which the performance data represents the performance and one or more speech periods in which the audio data represents the speech audio.

[0007] One aspect of the present invention relates to a program that enables a computer system to function as the following functional units: a data acquisition unit that acquires performance data representing a performer's performance on an instrument and audio data representing speech audio related to the performer's performance; and a display control unit that displays a reference screen on a display device, the reference screen displaying, based on a common timeline, one or more performance periods containing the performance data and one or more speech periods containing the speech audio represented by the audio data. Attached Figure Description

[0008] Figure 1 This is a block diagram illustrating the structure of the information processing system according to the first embodiment.

[0009] Figure 2 This is a block diagram illustrating the structure of a terminal device.

[0010] Figure 3 This is a block diagram illustrating the structure of a keyboard musical instrument.

[0011] Figure 4 This is a block diagram illustrating the structure of a control system.

[0012] Figure 5 This is a block diagram illustrating the functional structure of the control system.

[0013] Figure 6 This is a flowchart of the processing performed by the data acquisition department.

[0014] Figure 7 This is a diagram illustrating the collected data.

[0015] Figure 8 This is a diagram illustrating data management.

[0016] Figure 9 This is a diagram with reference to the image and detailed information.

[0017] Figure 10 It is a flowchart of the control process.

[0018] Figure 11 This is a schematic diagram of the image displayed by the display device of the terminal device.

[0019] Figure 12 This is a reference screen and schematic diagram showing details of the second embodiment.

[0020] Figure 13 This is a reference screen and schematic diagram showing details of the third embodiment.

[0021] Figure 14 This is a schematic diagram of the image displayed by the display device of the terminal device in the fourth embodiment.

[0022] Figure 15 This is a schematic diagram of the image displayed by the display device of the terminal device in the fifth embodiment.

[0023] Figure 16 This is a schematic diagram of a reference screen for the sixth embodiment.

[0024] Figure 17 This is a schematic diagram of the image displayed by the display device of the terminal device in the seventh embodiment.

[0025] Figure 18This is a schematic diagram of the image displayed by the display device of the terminal device in the seventh embodiment.

[0026] Figure 19 This is a schematic diagram of a reference screen for the seventh embodiment.

[0027] Figure 20 This is a schematic diagram of the image displayed by the display device of the terminal device in the eighth embodiment.

[0028] Figure 21 This is a schematic diagram of the image displayed by the display device of the terminal device in the 9th embodiment.

[0029] Figure 22 This is a schematic diagram of the image displayed by the display device of the terminal device in the 9th embodiment. Detailed Implementation

[0030] A: Implementation Method 1

[0031] Figure 1 This is a block diagram illustrating the structure of the information processing system 100 according to the first embodiment. The information processing system 100 is a computer system (i.e., a performance recording system or a performance management system) for recording and managing the performance of a player Ua on a keyboard instrument 10. The player Ua is, for example, a student practicing playing the keyboard instrument 10. The player Ua is, for example, a student in a music classroom. In addition to the player Ua, an instructor Ub is also present in the music classroom. The instructor Ub instructs the player Ua on playing the keyboard instrument 10.

[0032] The information processing system 100 includes a keyboard instrument 10, a recording system 20, a terminal device 30, a measurement system 40, and a control system 50. The terminal device 30 is, for example, an information device such as a smartphone, tablet computer, or personal computer.

[0033] Figure 2 This is a block diagram illustrating the structure of the terminal device 30. The terminal device 30 includes a control device 31, a storage device 32, a communication device 33, a display device 34, an operation device 35, a sound playback device 36, and a sound pickup device 37. Furthermore, the terminal device 30 can be implemented not only as a single device, but also as multiple devices configured separately from each other.

[0034] The control device 31 is composed of one or more processors of the various elements of the control terminal device 30. For example, the control device 31 is composed of one or more processors such as CPU (Central Processing Unit), GPU (Graphics Processing Unit), SPU (Sound Processing Unit), DSP (Digital Signal Processor), FPGA (Field Programmable Gate Array), or ASIC (Application Specific Integrated Circuit).

[0035] Storage device 32 is a single or multiple memory devices that store the program executed by control device 31 and various data used by control device 31. Storage device 32 is constructed from known recording media such as magnetic recording media or semiconductor recording media. Storage device 32 may also be constructed from a combination of various recording media. In addition, a removable recording medium that can be detached from terminal device 30, or a recording medium that can be written to or read from by control device 31 via communication network 200 (e.g., network hard drive) may be used as storage device 32.

[0036] The communication device 33 communicates with peripheral devices (such as the keyboard instrument 10, the recording system 20, the measuring system 40, and the control system 50). The communication device 33 communicates with the peripheral devices wirelessly, for example, via Wi-Fi (registered trademark) or Bluetooth (registered trademark). Communication between the communication device 33 and the peripheral devices can also be wired. The communication device 33 communicates with the control system 50, for example, via a communication network 200 such as the Internet.

[0037] The display device 34 displays images based on the control of the control device 31. The display device 34 may be composed of a display panel such as a liquid crystal panel or an organic EL (electroluminescence) panel. The operating device 35 is an input device that accepts operations from the user U (performer Ua or instructor Ub) of the terminal device 30. For example, an operating element operated by the user U or a touch panel that detects the user U's contact is used as the operating device 35. Furthermore, the display device 34 or the operating device 35, which are separate from the terminal device 30, may also be connected to the terminal device 30 via wired or wireless means. Moreover, the user U is not limited to performer Ua and instructor Ub. For example, the guardian of performer Ua may also be included in the user U.

[0038] The sound playback device 36 emits sound waves under the control of the control device 31. The sound playback device 36 may be an output device such as a loudspeaker or headphones. The sound playback device 36 may include a D / A converter that converts the audio signal of the playback object from digital to analog and an amplifier that amplifies the audio signal. Furthermore, the sound playback device 36, which is separate from the terminal device 30, can also be connected to the terminal device 30 via wired or wireless means.

[0039] The pickup device 37 generates an acoustic signal (hereinafter referred to as a "pickup signal") by picking up ambient sounds. Specifically, the sound produced by the keyboard instrument 10 through the performance of the player Ua is picked up by the pickup device 37. That is, the pickup signal is a signal representing the waveform of the sound produced by the keyboard instrument 10. For example, a single microphone or a microphone array with multiple microphones is shown as the pickup device 37. Furthermore, for convenience, illustrations of an A / D converter that converts the pickup signal from analog to digital and an amplifier that amplifies the pickup signal are omitted. The pickup device 37, which is separate from the terminal device 30, can also be connected to the terminal device 30 via wired or wireless means. The control device 31 stores the pickup signal generated by the pickup device 37 in the storage device 32.

[0040] Figure 3 This is a block diagram illustrating the structure of a keyboard instrument 10. The keyboard instrument 10 is an instrument that receives performance from a performer Ua, and includes a control device 11, a storage device 12, a communication device 13, a display device 14, an operation device 15, a sound source device 16, a sound output device 17, and a playing mechanism 18. The keyboard instrument 10 is an example of a playing device used by the performer Ua during performance. Furthermore, some or all of the functions of the terminal device 30 may also be incorporated into the keyboard instrument 10.

[0041] The playing mechanism 18 includes a keyboard 181, a sounding mechanism 183, a pedal 184, and a detection device 185. The keyboard 181 is a mechanism with multiple keys 182 arranged including white and black keys. Each key 182 is a playing operation that receives input from the player Ua (e.g., pressing and releasing a key). The sounding mechanism 183 is a striking mechanism that produces a sound by vibrating a string in conjunction with the operation of each key 182 on the keyboard 181. Specifically, the sounding mechanism 183 includes, for example, a string as a sound source, a hammer that strikes the string by rotating, and a transmission mechanism (e.g., a linkage rod, a push rod, or a repeating rod) that rotates the hammer in conjunction with the displacement of each key 182. Additionally, the sounding mechanism 183 has a mute mechanism that prevents the striking of the string in mute mode.

[0042] Pedal 184 is a performance control device operated by the performer Ua with their foot. For example, pedal 184 is a damper pedal (sustain pedal) used to indicate the sustain of a performance note, or a soft pedal used to reduce the volume of a performance note.

[0043] The detection device 185 is a sensor that detects the operation of each key 182 and pedal 184. Specifically, the detection device 185 detects the displacement of each key 182 and pedal 184. The detection device 185 may be composed of, for example, optical, mechanical, or magnetic sensors.

[0044] The control device 11 consists of one or more processors that control various elements of the keyboard instrument 10. For example, the control device 11 consists of one or more processors such as CPU, GPU, SPU, DSP, FPGA, or ASIC.

[0045] The control device 11 generates performance data D based on the detection results of the detection device 185. Performance data D represents the musical data of the performer Ua's performance on the keyboard instrument 10. Specifically, performance data D is, for example, a time sequence of event data according to the MIDI (Musical Instrument Digital Interface) standard. That is, performance data D specifies the pitch (note number) and intensity (tempo) of the key 182 operated by the performer Ua in a time sequence. Additionally, performance data D also represents the performer Ua's operation of the pedal 184. As described above, the control device 11 functions as a music data generation unit that generates performance data D based on the detection results of the detection device 185.

[0046] The display device 14 displays images based on the control of the control device 31. The display device 14 is, for example, a display panel such as a liquid crystal panel or an organic EL panel. The operating device 15 is an input device that accepts operations from the player Ua. For example, an operating element operated by the player Ua or a touch panel that detects the player Ua's contact is used as the operating device 15. In addition, the display device 14 or the operating device 15, which are separate from the keyboard instrument 10, can also be connected to the keyboard instrument 10 via wired or wireless means.

[0047] The sound source device 16 generates an acoustic signal representing a played tone corresponding to the detection result of the detection device 185. Specifically, the sound source device 16 generates the acoustic signal based on the playing data D generated by the control device 11. For example, it generates an acoustic signal corresponding to the pitch of the key 182 whose displacement was detected by the detection device 185. Furthermore, the control device 11 can also implement the function of the sound source device 16 by executing a program stored in the storage device 12. As described above, the keyboard instrument 10 is an electronic instrument capable of selectively producing natural tones produced by the sound-producing mechanism 183 and electronic tones produced by the sound source device 16. Alternatively, either the sound-producing mechanism 183 or the sound source device 16 may be omitted.

[0048] The sound output device 17 emits sound waves under the control of the control device 11. The sound output device 17 is, for example, an output device such as a speaker or headphones. Specifically, the sound output device 17 emits performance sounds represented by the acoustic signals generated by the sound source device 16. As explained above, the keyboard instrument 10 of the first embodiment can selectively perform the generation of performance sounds implemented by the sound-producing mechanism 183 and the generation of performance sounds implemented by the sound source device 16. Furthermore, the sound output device 17 may also include a D / A converter that converts the acoustic signals from digital to analog and an amplifier that amplifies the acoustic signals. Moreover, the sound output device 17, which is separate from the keyboard instrument 10, can also be connected to the keyboard instrument 10 via wired or wireless means.

[0049] Storage device 12 is one or more memory devices that store the program executed by control device 11 and various data used by control device 11. Storage device 12 is constructed from known recording media such as magnetic recording media or semiconductor recording media. Storage device 12 may also be constructed from a combination of various recording media. In addition, a removable recording medium that is detachable from keyboard instrument 10, or a recording medium that can be written to or read from by control device 11 via communication network 200 (e.g., network hard drive), may also be used as storage device 12.

[0050] Additionally, storage device 12 stores performance information X. Performance information X is information related to the performance of performer Ua and is temporarily stored on keyboard instrument 10 until it is transferred to terminal device 30. That is, after being transferred to terminal device 30, performance information X is deleted from storage device 12. Performance information X also represents information (performance log) indicating the performance history of performer Ua. Specifically, performance information X includes performer Ua's identification information, the sound source used, the usage mode, the performance date and time (start time and end time), the performance location, the performance scene, the identification information and model of keyboard instrument 10, the name of the teaching sheet music used by performer Ua, and the sheet music information of the piece performed by performer Ua. In addition, performance information X includes the aforementioned performance data D indicating the performance of performer Ua.

[0051] The communication device 13 communicates with the terminal device 30. The communication device 13 communicates with the terminal device 30 wirelessly, for example, via Wi-Fi (registered trademark) or Bluetooth (registered trademark). For example, the communication device 13 sends performance information X stored in the storage device 12 to the terminal device 30. Furthermore, the communication between the communication device 13 and the terminal device 30 can also be wired. The communication device 13, which is separate from the keyboard instrument 10, can also be connected to the keyboard instrument 10 via wired or wireless means.

[0052] Figure 1The recording system 20 records the performance of performer Ua. Specifically, the recording system 20 has a camera device 21 and a microphone device 22 installed in the music classroom. Furthermore, the camera device 21 can also be mounted on the terminal device 30 or the keyboard instrument 10. Similarly, the microphone device 22 can also be mounted on the terminal device 30 or the keyboard instrument 10. Additionally, the microphone device 22 can also be mounted on the camera device 21. The recording system 20 may also include multiple camera devices 21 or multiple microphone devices 22.

[0053] The shooting device 21 is a camera that generates image data V by capturing the performer Ua and the keyboard instrument 10. Image data V is data of moving images obtained from the performance of the keyboard instrument 10 by the performer Ua. The format of image data V is arbitrary, for example, MPEG (moving picture experts group) format is exemplified. Specifically, the shooting device 21 includes multiple cameras that capture the performer Ua from different directions. Therefore, image data V includes multiple moving images obtained from the performance of the keyboard instrument 10 by the performer Ua from different directions (e.g., see reference). Figure 12 (Partial image E3).

[0054] The shooting device 21 includes, for example, an optical system such as a camera lens, a shooting element that receives incident light from the optical system, and a processing circuit that generates image data V based on the amount of light received by the shooting element. Furthermore, in addition to dedicated imaging equipment, general-purpose information devices equipped with shooting functions (such as smartphones or tablets) can also be used as the shooting device 21. Moreover, the shooting device 21 can also begin shooting in conjunction with the performer Ua starting to play the keyboard instrument 10.

[0055] Figure 1 The pickup device 22 is one or more microphones that pick up sounds around the performer Ua. The pickup device 22 generates audio data A representing the waveform of the surrounding sounds. The audio data A can be in any form, for example, in wav format.

[0056] Specifically, in addition to the playing sounds produced by the keyboard instrument 10 through the performance of the performer Ua, the sound pickup device 22 also records audio of the instructor Ub speaking (hereinafter referred to as "speech audio"). The speech audio of the instructor Ub is audio related to the performance of the performer Ua. For example, as speech audio, examples include various audio used to guide the performer Ua in playing the keyboard instrument 10, such as explanations of playing methods, evaluations of the quality of the performance, and criticisms of problems related to the performance.

[0057] Furthermore, the pickup device 37 of the terminal device 30 can also generate audio data A of spoken audio in the same way as the pickup device 22. In addition, besides devices dedicated to pickup, general information devices equipped with pickup functions (such as smartphones or tablets) can also be used as the pickup device 22. The pickup device 22 can also start pickuping in conjunction with the performer Ua starting to play the keyboard instrument 10.

[0058] The imaging device 21 is connected to the terminal device 30 via wired or wireless means. The image data V generated by the imaging device 21 is sent to the terminal device 30. Similarly, the audio pickup device 22 is connected to the terminal device 30 via wired or wireless means. The audio data A generated by the audio pickup device 22 is sent to the terminal device 30.

[0059] Figure 1 The measurement system 40 measures biometric information Y related to the performer Ua. Specifically, the measurement system 40 has a biosensor 41 and a center of gravity sensor 42. The biosensor 41 is a movable sensor worn on the body of the performer Ua. For example, the biosensor 41 is a heart rate sensor that detects the performer Ua's heart rate. The center of gravity sensor 42 is a sensor installed on the chair in which the performer Ua sits when playing the keyboard instrument 10, and detects the movement of the center of gravity of the seat surface. The biometric information Y of the performer Ua includes data such as the performer Ua's playing posture, tension (tension), heart rate, and concentration.

[0060] Playing posture is an indicator related to the posture of the player Ua who is playing the keyboard instrument 10. The playing posture is determined based on the analysis of the detection results of the center of gravity sensor 42. Alternatively, the playing posture can also be determined by analyzing the image of the player Ua captured by the imaging device 21.

[0061] Heart rate is detected, for example, via a heart rate sensor. Tension is an indicator of the performer Ua's level of tension (tension), calculated, for example, based on the performer Ua's heart rate and the stability of their playing posture. Concentration is an indicator of the performer Ua's focus on the performance, calculated, for example, based on the performer Ua's heart rate and the stability of their playing posture. Alternatively, the concentration value can be set to a higher value for the number of keystrokes within a specified time. Furthermore, the structure and type of the measurement system 40 are not limited to the examples above.

[0062] Furthermore, the structure used to obtain bio-information Y is not limited to the movable biosensor 41 worn on the body of the performer Ua. For example, bio-information Y of the performer Ua can be generated by analyzing image data V generated by an imaging device such as the imaging device 21 through imaging of the performer Ua. For the generation of bio-information Y achieved by analyzing image data V, for example, the technology described in Japanese Patent Application Publication No. 2021-069813 can be used.

[0063] In the structure described above, the terminal device 30 obtains various information from peripheral devices (keyboard instrument 10, recording system 20, and measurement system 40). For example, the control device 31 sends an information request from the communication device 33 to the peripheral devices, and the communication device 33 receives the information sent from the peripheral devices according to the information request. Specifically, the terminal device 30 obtains performance information X, including performance data D, from the keyboard instrument 10, obtains image data V and audio data A from the recording system 20, and obtains biometric information Y of the performer Ua from the measurement system 40.

[0064] The recording data R1 is stored in the storage device 32 of the terminal device 30. The recording data R1 is comprehensive data related to the performance of the performer Ua. Specifically, the recording data R1 includes performance information X, video data V, audio data A, and biological information Y. For each practice session of the performer Ua (e.g., each lesson in the music classroom), the recording data R1 is stored in the storage device 32. The control device 31 sends the recording data R1 from the communication device 33 to the control system 50.

[0065] The control system 50 receives the recorded data R1 sent from the terminal device 30. The control system 50 is a computer system for managing the recorded data R1. Furthermore, in the first embodiment, a structure is shown where the recorded data R1 is sent to the control system 50 via the terminal device 30; however, the recorded data R1 can also be sent to the control system 50 without going through the terminal device 30. For example, performance information X can be sent from the keyboard instrument 10 to the control system 50, image data V can be sent from the imaging device 21 to the control system 50, audio data A can be sent from the sound pickup device 22 to the control system 50, and biological information Y can be sent from the measurement system 40 to the control system 50.

[0066] Figure 4 This is a block diagram illustrating the structure of the control system 50. The control system 50 includes a control device 51, a storage device 52, and a communication device 53. Furthermore, the control system 50 can be implemented not only as a single device, but also as multiple devices configured as separate units.

[0067] The control device 51 consists of one or more processors that control various elements of the keyboard instrument 10. For example, the control device 51 consists of one or more processors such as CPU, GPU, SPU, DSP, FPGA, or ASIC.

[0068] Storage device 52 is one or more memory devices that store the program executed by control device 51 and various data used by control device 51. Storage device 52 is constructed from known recording media such as magnetic recording media or semiconductor recording media. Storage device 52 may also be constructed from a combination of various recording media. In addition, a removable recording medium that can be detached from control system 50, or a recording medium that can be written to or read from by control device 51 via communication network 200 (e.g., network hard drive) may also be used as storage device 52.

[0069] Communication between communication device 53 and terminal device 30 is performed via communication network 200. Furthermore, communication between communication device 53 and terminal device 30 can also be wireless communication, such as Wi-Fi (registered trademark) or Bluetooth (registered trademark).

[0070] Figure 5 This is a block diagram illustrating the functional structure of the control system 50. The control device 51 executes the program stored in the storage device 52, thereby realizing multiple functions for managing the recorded data R1 (recorded data acquisition unit 511, information processing unit 512, performance evaluation unit 513, display control unit 514, selection receiving unit 515, and information output unit 516).

[0071] The data acquisition unit 511 acquires the data R1. That is, the data acquisition unit 511 acquires the performance information X (performance data D), the image data V, the audio data A, and the biological information Y. Figure 6 This is a flowchart of the processing performed by the data acquisition unit 511. If processing begins, the data acquisition unit 511 receives the data R1 (Sa1) sent from the terminal device 30 via the communication device 53.

[0072] The data acquisition unit 511 manages the temporal correspondence (Sa2) of the performance information X (performance data D), video data V, audio data A, and biological information Y of the recorded data R1. Specifically, the data acquisition unit 511 adds synchronization data to each of the performance data D, video data V, audio data A, and biological information Y. Synchronization data is data that represents the time points (hereinafter referred to as "reference time points") that correspond in time between the performance data D, video data V, and audio data A. For example, LTC (Longitudinal Time Code) or MTC (MIDI Time Code) are examples of synchronization data.

[0073] Furthermore, the time shown in the synchronization data may be, for example, Coordinated Universal Time (UTC) or Japan Standard Time (JTC), or it may be the time inherent to the information processing system 100. In the case of using a separate time, for example, the terminal device 30 specifies the time as starting from the time it logs into the control system 50.

[0074] Alternatively, synchronization data can be generated in peripheral devices of the terminal device 30 (keyboard instrument 10, imaging device 21, sound pickup device 22, measurement system 40). For example, the keyboard instrument 10 assigns MTC as synchronization data to the performance data D. The recording data acquisition unit 511 assigns synchronization data shared with the performance data D to data other than the performance data D (image data V, audio data A, biological information Y). Furthermore, synchronization data can be shared between peripheral devices.

[0075] The data acquisition unit 511 generates data R2 (Sa3) containing data R1, and stores data R2 in the storage device 52 (Sa4). Furthermore, data R1 and data R2 are examples of "data" in this invention.

[0076] Figure 7 This is a schematic diagram of the recorded data R2. The recorded data R2 is obtained by appending the user identification information Fa, device identification information Fb, and information category Fc to the performance data D, video data V, and audio data A contained in the recorded data R1, respectively.

[0077] User identification information Fa is a string of symbols used to identify the performer Ua, who is included in the recording object involved in the recording data R1. For example, the identification information contained in the performance information X is registered as user identification information Fa in the recording data R2. In addition, the identification information of, for example, the terminal device 30 can also be used as user identification information Fa. Furthermore, user identification information Fa can be changed according to instructions from user U.

[0078] Device identification information Fb is a symbol used to identify the device used in the recording. For example, for performance data D, the identification information of the keyboard instrument 10 is attached ( Figure 7 The “Piano#1” is used as the device identification information Fb. The identification information of the imaging device 21 is added to the image data V. Figure 7 The “Cam#1” is used as the device identification information Fb. The identification information of the pickup device 22 is added to the audio data A. Figure 7The device identification information Fb is defined as "Mic#1". Furthermore, the device identification information Fb can be included in the data acquisition R1 or generated by the data acquisition unit 511. Additionally, the device identification information Fb can be changed based on instructions from the user U.

[0079] Information category Fc is the category of information for each of the performance data D, video data V, and audio data A. For example, the performance data D may be assigned the information category Fc of "performance," the video data V may be assigned the information category Fc of "video," and the audio data A may be assigned the information category Fc of "audio." The data acquisition unit 511 identifies the information category Fc by referring to, for example, the metadata attached to each of the performance data D, video data V, and audio data A. Furthermore, the identification of information category Fc may also utilize, for example, the identification process described below, which utilizes statistical models such as deep neural networks or HMMs (Hidden Markov Models).

[0080] Figure 5 The information processing unit 512 generates management data R3 based on the collected data R2. Management data R3 is used to manage the performance of performer Ua in terms of time. Figure 8 This is a schematic diagram of management data R3. Management data R3 is a set of multiple unit data Z. Each of the multiple unit data Z corresponds to any one of the performance data D, image data V, and audio data A.

[0081] The performer Ua does not play the keyboard instrument 10 continuously, but intermittently at appropriate intervals. For example, the performer Ua stops playing while the instructor Ub is speaking. The unit data Z corresponding to the performance data D is data used to manage the period of performance represented by the performance data D on the time axis (hereinafter referred to as "performance period"). The performance period is the period during which the performer Ua actually plays the keyboard instrument 10 (i.e., the period excluding the period during which the performer Ua stops playing). For example, each period adjacent on the time axis at intervals greater than a predetermined length is divided into performance periods. That is, intervals less than a predetermined length during the performance of the performer Ua are included in one performance period. Furthermore, the predetermined length applied to the division of performance periods (i.e., the threshold related to the interval of performance periods) can be a fixed length set in advance, or a variable length arbitrarily set by the user U, for example. In addition, if the duration of a performance period is less than a predetermined threshold (e.g., 1 second), the information processing unit 512 can delete the data within that performance period in the performance data D as noise data. That is, data within a very short performance period in the performance data D can also be excluded from the storage object for the storage device 52.

[0082] The unit data Z corresponding to the performance data D can be generated for each performance period in the performance by performer Ua. Therefore, the unit data Z is generated corresponding to the number of performance periods represented by the performance data D.

[0083] In addition to user identification information Fa, device identification information Fb, and information category Fc, each unit data Z corresponding to performance data D also includes period data T and performance data Qd (Qd1, Qd2, ...). Period data T is a timestamp specifying the performance period. Specifically, period data T specifies the start and end times of the performance period, which begins at a reference time point represented by synchronization data. Performance data Qd is the portion of the performance period specified by period data T within performance data D. That is, performance data Qd represents the time-series data of performer Ua's performance during the performance period. As understood from the above explanation, performance data Qd, obtained by dividing performance data D by performance period, is included in each unit data Z. Furthermore, each unit data Z corresponding to performance data D may contain the sheet music title of the piece performed by performer Ua. The sheet music title is included in performance information X.

[0084] Furthermore, the instructor Ub does not speak continuously to the performer Ua, but rather intermittently at appropriate intervals. For example, the instructor Ub may stop speaking while the performer Ua is playing. The unit data Z corresponding to the audio data A is data used to manage the period (hereinafter referred to as "speech period") during which the audio data A represents the instructor Ub's speech audio exists on the timeline. That is, the speech period is the period during which the instructor Ub actually speaks. For example, each period adjacent to the other on the timeline at an interval greater than a predetermined length is divided into a speech period. That is, intervals less than a predetermined length during the instructor Ub's speech are included in one speech period. In addition, the predetermined length applied to the division of speech periods (i.e., the threshold related to the interval of speech periods) can be a fixed length set in advance, or it can be a variable length arbitrarily set by the user U, for example. Furthermore, if the duration of a speech period is less than a predetermined threshold (e.g., 1 second), the information processing unit 512 may also delete the data within that speech period in the audio data A as noise data. That is, data within a very short speaking period in audio data A can also be excluded from the storage object for storage device 52.

[0085] The unit data Z corresponding to the audio data A is generated for each speaking period of the instructor Ub's speech. Therefore, the unit data Z is generated corresponding to the number of speaking periods of the speech represented by the audio data A.

[0086] In addition to user identification information Fa, device identification information Fb, and information category Fc, each unit data Z corresponding to audio data A also includes period data T and audio data Qa (Qa1, Qa2, ...). Period data T is a timestamp specifying the speech period. Specifically, period data T specifies the start and end times of the speech period, which is based on a reference time point represented by synchronized data. Audio data Qa is the portion of audio data A within the speech period specified by period data T. That is, audio data Qa is the time-series data representing the speech of the instructor Ub during the speech period. As understood from the above explanation, audio data Qa, obtained by dividing audio data A into speech periods, is included in each unit data Z.

[0087] As explained above, the information processing unit 512 generates period data T for specifying the performance period based on the performance data D, and generates period data T for specifying the speech period based on the audio data A.

[0088] Figure 5 The performance evaluation unit 513 evaluates the performance of performer Ua represented by performance data D. Specifically, the performance evaluation unit 513 evaluates the quality of performer Ua's performance by comparing the performance represented by performance data D with a demonstrative performance represented by reference data (hereinafter referred to as "demonstration performance"). The reference data is the music data representing the score of the piece. Specifically, the reference data is, for example, a time sequence of event data according to the MIDI standard. The performance evaluation unit 513 arbitrarily employs known techniques to evaluate the performance.

[0089] The performance evaluation unit 513 determines the period in the performance represented by the performance data D that approximates the demonstration performance (hereinafter referred to as the "demonstration period"), and generates period data T that specifies the demonstration period. The demonstration period is the period during which the performer Ua performed appropriately. For example, a period in which the evaluation value corresponding to the difference between the performer Ua's performance and the demonstration performance is greater than a predetermined threshold is determined as the demonstration period. The period data T specifies the start and end times of the demonstration period, which starts from a reference time point represented by the synchronization data. Furthermore, in the following description, the performance period, the speech period, and the demonstration period are collectively referred to as the "object period".

[0090] In addition, the performance evaluation unit 513 determines the periods of high difficulty in the performance represented by the reference data (hereinafter referred to as "high difficulty periods"), and generates period data T for specifying the high difficulty periods. High difficulty periods are, for example, periods in which the number of notes per unit time exceeds a specified threshold (e.g., periods with high note density due to chords or legato), periods with a wide range of notes, periods requiring a wide range of fingering, periods in which modulation occurs, etc.

[0091] The display control unit 514 displays various images on the display device 34 of the terminal device 30. Specifically, the display control unit 514 displays the image on the display device 34 of the terminal device 30 by sending image data representing the image to the terminal device 30. For example, the image instructed by the display control unit 514 is displayed on the terminal device 30 used by the performer Ua or the terminal device 30 used by the instructor Ub.

[0092] The display control unit 514 of the first embodiment displays a reference screen 61 on the display device 34 of the terminal device 30. The reference screen 61 is an image showing the result of analyzing the performance of the performer Ua. Figure 9 This is a schematic diagram of reference screen 61. Reference screen 61 includes the subject's name 611, the musical score image 612, and multiple regions G (G1 to G4) corresponding to different types of the subject. The subject's name 611 is the name of the performer Ua, "UserA".

[0093] Referencing screen 61, a timeline τ is set. Timeline τ represents a series of performance periods for the performer Ua. Specifically, timeline τ corresponds, for example, to the duration of one lesson in a music classroom. That is, for example, the start point of a lesson in the music classroom corresponds to the start point of timeline τ, and the end point of that lesson corresponds to the end point of timeline τ. Furthermore, timeline τ can also correspond, for example, to the login period of terminal device 30. That is, for example, the login time of terminal device 30 corresponds to the start point of timeline τ, and the logout time of terminal device 30 corresponds to the end point of timeline τ.

[0094] An indicator 613 is set on the time axis τ. Indicator 613 is a pointer that designates a period during the performance of performer Ua that becomes the display object of reference screen 61. The score image 612 is the score of a piece of music performed by performer Ua, including the time point indicated by indicator 613, within a defined interval. User U can use the operating device 35 to move indicator 613 along the time axis τ. The score image 612 corresponding to the position of indicator 613 on the time axis τ is displayed on reference screen 61. Therefore, the time axis of score image 612 and the time axis τ of reference screen 61 have different scales.

[0095] High-difficulty images 614 are displayed in the score image 612. High-difficulty images 614 represent high-difficulty periods in the piece of music represented by score image 612, and are displayed for each high-difficulty period.

[0096] Multiple regions G are each rectangular regions that extend along the time axis τ. These regions G are arranged in a direction orthogonal to the time axis τ. The time axis τ is shared by the multiple regions G. The multiple regions G include a performance region G1, an evaluation region G2, a speech region G3, and an image region G4.

[0097] The performance area G1 is the area displaying the performance period. Specifically, the performance area G1 displays the display item name N1 and the period image B1. The display item name N1 is the name of the displayed object, "Performance". The period image B1 is an image representing the performance period in which performer Ua is playing, displayed for each performance period. The position and display length of the period image B1 along the time axis τ are set according to the start and end times specified by the period data T.

[0098] Evaluation area G2 is the area displaying the demonstration period. Specifically, evaluation area G2 displays the display item name N2 and the period image B2. The display item name N1 is the name of the displayed object, "Demonstration Performance". The period image B2 is an image representing the demonstration period in which the performer Ua performed appropriately, displayed for each demonstration period. The position and display length of the period image B1 along the time axis τ are set according to the start and end times specified by the period data T of the demonstration period.

[0099] The speech area G3 is the area displaying the speech period. Specifically, speech area G3 displays the item name N3 and the period image B3. The item name N3 is the name of the displayed object, "Teacher's Explanation". The period image B3 is an image representing the speech period of the instructor Ub, displayed for each speech period. The position and display length of the period image B3 along the time axis τ are set according to the start and end times specified by the period data T.

[0100] Image region G4 is the area where the image data V is displayed sequentially along the time axis τ. The display item name N4 is displayed within image region G4. Display item name N4 is the name of the displayed object, "Image".

[0101] Each period image B (B1 to B3) is displayed in a different display mode. The display mode refers to the characteristics of the image that the user U can visually distinguish. Examples of display modes include display color, pattern, size, or shape. Display color is defined by hue (tone), chroma, or brightness (grayscale). The display mode of each period image B changes according to instructions from the user U to the operating device 35. For example, the user U can select the display mode of the period image B in a given area G from multiple candidates displayed on the display device 34 by selecting the display item name N (N1 to N4) within that area G. Furthermore, the string or display mode of the display item name N (N1 to N4) in each area G can also be changed, for example, based on operations performed on the operating device 35.

[0102] As illustrated above, reference screen 61 is an image that displays the performance period (G1), demonstration period (G2), speech period (G3), and image data V based on a common time axis τ. For example, the performance period (G1) and speech period (G3) are displayed based on the common time axis τ. Therefore, according to the first embodiment, the user U can visually and intuitively grasp the approximate relationship between the performance period in which the performer Ua performs and the speech period in which the speech audio associated with the performance is present on the time axis τ.

[0103] Additionally, the display control unit 514 displays the reference screen 61 using the period data T. That is, it generates period data T that specifies the performance period and the speaking period, and displays the reference screen 61 using the period data T. The reference screen 61 also displays an index related to the performance of the performer Ua.

[0104] Furthermore, in the first embodiment, the demonstration period corresponding to the performance that approximates the demonstration performance is displayed based on the timeline τ shared with the performance period and the speaking period. Therefore, the user U can visually and intuitively grasp the appropriate performance period in the performance of the performer Ua.

[0105] User U operates the operation device 35 while visually confirming the reference screen 61 displayed on the display device 34, thereby selecting the desired object period among multiple object periods (performance period, speech period, demonstration period) within the reference screen 61. For example, user U can specify any of the multiple period images B within the reference screen 61 by clicking, thereby selecting the object period corresponding to that period image B. Figure 5 The selection receiving unit 515 accepts the user U's selection of any one of the multiple object periods in the reference screen 61. That is, the selection receiving unit 515 accepts the selection of any one of the multiple performance periods (G1), any one of the multiple demonstration periods (G2), or any one of the multiple speaking periods (G3).

[0106] When the selection receiving unit 515 receives a selection of a target period, the information output unit 516 outputs detailed information E related to that target period. For example, if the user U selects period image B1 or period image B2, the information output unit 516 will output detailed information E related to that target period. Figure 9 As illustrated, the image representing a portion of the performance data D represented by the period image B (hereinafter referred to as "performance image E1") is displayed on the display device 34 as detailed information E. That is, each period image B of the reference screen 61 functions as an index for the user U to select the period to which detailed information E should be displayed. Figure 9 The detailed information E (E1, E2) is shown together with reference screen 61.

[0107] For example, when period image B1 is selected, the information output unit 516 displays performance image E1, which represents the portion of the performance data D corresponding to period image B1 within the performance period, as detailed information E on the display device 34. Performance image E1 is a piano scroll image representing performance data Qd. That is, performance image E1, which displays note images (note bars) representing each note in an area with a pitch axis (horizontal axis) and a time axis (vertical axis), is displayed as detailed information E on the display device 34.

[0108] Similarly, when period image B2 is selected, the information output unit 516 displays the performance image E1, representing the portion of the performance data D within the demonstration period corresponding to period image B2, as detailed information E on the display device 34. Furthermore, when either period image B1 or period image B2 is selected, the image represented by the image data V can also be displayed in the image area G4.

[0109] Furthermore, when period image B3 is selected, the information output unit 516 plays a portion of the audio data A corresponding to the speech period of period image B3 (hereinafter referred to as "partial audio E2") from the audio playback device 36. Specifically, the information output unit 516 sends the audio data Qa of the speech period corresponding to period image B3 selected by the user U to the terminal device 30. Therefore, the partial audio E2 of the instructor Ub during the speech period is played out from the audio playback device 36 as detailed information E.

[0110] As described above, the output (display or playback) is the detailed information E corresponding to the period image B selected by the user U from among multiple period images B. Therefore, the user U can confirm the details of the performer Ua's performance for the desired period among multiple periods (performance period, demonstration period, speech period, video period).

[0111] Figure 10 This is a flowchart of the process (hereinafter referred to as "control process") performed by the control device 51. Figure 11 This is a schematic diagram of an image displayed on the display device 34 of the terminal device 30 in conjunction with the progress of control processing. For example, control processing begins upon receiving a login request sent by the terminal device 30. The terminal device 30 can establish communication with the control system 50 by logging into the control system 50.

[0112] To log in to the control system 50, the control device 51 displays an authentication screen 621 (Sb1) on the display device 34 of the terminal device 30. The authentication screen 621 is, for example, a screen for inputting authentication information for the user U. The authentication information includes, for example, identification information and a password. Furthermore, the user U is only required to input authentication information upon initial login; subsequent logins can omit this step.

[0113] The authentication process can be biometric authentication that utilizes the biometric information of the user U. Biometric information could be, for example, a facial image of the user U (facial authentication) or an audio signal (audio authentication).

[0114] For facial authentication, recognition techniques can be used, for example, those employing statistical models such as deep neural networks or Hidden Markov Models (HMMs). For instance, facial authentication can be performed using Haar-like features extracted from facial images. Alternatively, general-purpose software can also be used for facial authentication.

[0115] For audio authentication, recognition techniques can be used, for example, those employing statistical models such as deep neural networks or Hidden Markov Models (HMMs). For instance, after performing audio recognition using a statistical model, audio authentication can be performed using individual audio feature quantities. Alternatively, general-purpose software can also be used for audio authentication.

[0116] In addition, when the terminal device 30 starts referring to the display of the screen 61, it is also necessary to log in to the control system 50 from the terminal device 30 when sending various data related to the performance of the performer Ua to the control system 50.

[0117] The control device 51 displays the device setting screen 622 on the display device 34 (Sb2) of the terminal device 30. The device setting screen 622 is an image used by the user U to set the validity / invalidity of various peripheral devices. The user U can select validity / invalidity for each peripheral device by operating the operation device 35. The data of the peripheral devices set as valid in the device setting screen 622 becomes the display object of the reference screen 61.

[0118] Control device 51 Figure 11 As illustrated, the saved data list 623 is displayed on the display device 34 (Sb3) of the terminal device 30. The saved data list 623 is a list of previously saved data R1 by the user U. Specifically, the save date, time, and duration are displayed for each previously saved data R1. The user U can select any of the multiple saved data R1s displayed in the saved data list 623 by operating the operation device 35. The control device 51 (display control unit 514) selects management data R3 (Sb4) corresponding to the saved data R1 selected by the user U from the storage device 52. Using the management data R3 selected from the storage device 52, the control device 51 (display control unit 514) displays the reference screen 61 on the display device 34 (Sb5) of the terminal device 30.

[0119] The control device 51 (selection receiving unit 515) receives a selection from the user U of any of the multiple object periods (Sb6). The control device 51 (information output unit 516) outputs detailed information E corresponding to the object period selected by the user U via the terminal device 30 (Sb7). The operation of the control system 50 in the first embodiment is as described above.

[0120] B: Implementation Method 2

[0121] The second embodiment will be described. Furthermore, in the embodiments illustrated below, elements that function the same as those in the first embodiment will be referred to by the same reference numerals as those used in the description of the first embodiment, and detailed descriptions of each will be omitted as appropriate.

[0122] Figure 12 This is a schematic diagram of reference screen 61 and detailed information E of the second embodiment. Reference screen 61 of the second embodiment is the same as that of the first embodiment. The detailed information E of the second embodiment contains multiple pieces of information with different information categories Fc.

[0123] If the selection receiving unit 515 receives a selection from the user U for any of the multiple object periods (hereinafter referred to as "selection period"), the information output unit 516 selects at least one other object period on the time axis τ that overlaps with the selection period. Specifically, based on the user U's selection of one selection period, multiple object periods including that selection period are selected. That is, the information output unit 516 groups the multiple object periods, including the selection period selected by the user U and the object periods that overlap with that selection period.

[0124] The information output unit 516 displays the detailed information E corresponding to each of the multiple objects side-by-side on the display device 34. Figure 12The example illustrates a scenario where user U selects a performance period as the selected period. In this scenario, in addition to the performance image E1 of the performance period selected by user U, partial images E3 and performance parameter conversions E4 are also displayed as detailed information E on the display device 34. Partial image E3 is a partial image within the object period overlapping with the performance period in the image represented by image data V. Specifically, multiple dynamic images obtained by capturing the performance of the keyboard instrument 10 by performer Ua from different directions are displayed as partial images E3.

[0125] Performance parameter transformation E4 is a graph representing the temporal changes of performance parameters. Performance parameters are, for example, performance-related parameters represented by performance data D. For instance, the intensity (tempo) of the keystrokes by performer Ua is exemplified as a performance parameter. Furthermore, multiple detailed information E can also be displayed in the order of the time series specified by the period data T.

[0126] In the second embodiment, the same effect as in the first embodiment is achieved. Furthermore, in the second embodiment, detailed information E related to multiple object periods is output by selecting one object period. Therefore, compared to the method of individually selecting an object period for each desired detailed information E (e.g., in the first embodiment), the operational burden on the user U can be reduced.

[0127] C: Third Implementation

[0128] Figure 13 This is a schematic diagram of reference screen 61 and detailed information E of the third embodiment. Reference screen 61 of the third embodiment is the same as that of the first embodiment. If a selection of a speaking period is received from the user U, the information output unit 516 of the third embodiment displays detailed information E indicating the speaking content of the instructor Ub during that speaking period on the display device 34. Specifically, the information output unit 516 generates a guidance string E5, which indicates the content of the guidance for the performer Ua.

[0129] First, the information output unit 516 determines the speech string by audio recognition of audio data Qa corresponding to the speech period. The speech string is a string representing the content of the instructor Ub's speech. For audio recognition of audio data Qa, any known technique is arbitrarily employed. The information output unit 516 extracts a string from the speech string that includes a pre-defined specific statement (hereinafter referred to as "specific statement"). For example, it extracts a string from the speech string that includes specific statements such as "today's key points" or "homework".

[0130] Information output unit 516 retrieves a template string pre-stored in storage device 52. The template string is a template-type string representing instruction for performer Ua. The template string is a string where the portion corresponding to a specific statement is left blank. Information output unit 516 generates instruction string E5 by inserting strings extracted from the speech string into the blank spaces corresponding to specific statements in the template string. Therefore, instruction string E5 is a string summarizing the portion of the instructor Ub's speech audio, particularly those related to specific statements. Instruction string E5 is displayed as detailed information E on display device 34. Instruction string E5 is used by performer Ua as lesson notes related to the instructor Ub's instruction. As described above, instruction string E5 is a combination of the pre-prepared template string and a portion of the instructor Ub's speech string. Furthermore, instruction string E5 can also be edited according to instructions from user U to operation device 35.

[0131] Furthermore, the information output unit 516 generates edited animation E6 by combining portions of the video data V (represented by image data V) and the audio data A (represented by audio data A) selected by the user U within each object's duration (e.g., linking them in a temporally continuous manner). Edited animation E6 is a montage animation composed of video and audio representing the performance of the performer Ua. Additionally, the information output unit 516 can also include the interval in the video data V where the instructor Ub speaks a specific phrase within the edited animation E6. The information output unit 516 displays the edited animation E6 on the display device 34 of the terminal device 30.

[0132] In the third embodiment, the same effect as in the first embodiment is achieved. Furthermore, in the third embodiment, the instruction string E5 representing the instruction of the instructor Ub is displayed on the display device 34, so the user U can visually confirm the instruction of the instructor Ub. Additionally, since the editing animation E6 is displayed on the display device 34, the user U (performer Ua or instructor Ub) can visually confirm how the performer Ua is playing.

[0133] D: Implementation Method 4

[0134] Figure 14This is a schematic diagram of the image displayed on the display device 34 of the terminal device 30 that has logged into the control system 50. When the display device 34 displays the saved data list 623 (Sb3), and the user U operates the operation device 35, the control device 51 (display control unit 514) displays a viewing permission screen 624 on the display device 34. The viewing permission screen 624 is a list of multiple candidates associated with the user U. For example, it displays multiple candidates associated with a string entered by the user U. By operating the operation device 35, the user U can select one or more candidates for the permission display reference screen 61. That is, viewing the reference screen 61 is prohibited for candidates not selected by the user U.

[0135] If a candidate is selected, the control device 51 (display control unit 514) stores the identification information of the user U who has been granted permission to view the reference screen 61 in the storage device 52. The control device 51 (display control unit 514) displays a saved data list 623' containing the identification information of the user U who has been granted permission to view the reference screen 61 on the display device 34.

[0136] If a user U requests the display of reference screen 61 from its terminal device 30, the control device 51 (display control unit 514) determines whether to allow viewing of reference screen 61 for the user U. If viewing of reference screen 61 is allowed, the control device 51 (display control unit 514) displays reference screen 61 on the display device 34 of the terminal device 30, in the same manner as described above. On the other hand, if viewing of reference screen 61 is prohibited, the control device 51 (display control unit 514) displays a message indicating that viewing is prohibited on the display device 34. That is, reference screen 61 is not displayed on the display device 34.

[0137] In the fourth embodiment, the same effect as in the first embodiment is achieved. Furthermore, in the fourth embodiment, since permission to view the reference screen 61 is set for each user U, the scope of disclosure of each reference screen 61 can be limited.

[0138] E: Fifth Implementation

[0139] Figure 15This is a schematic diagram of the image displayed on the display device 34 of the terminal device 30 that has logged into the control system 50. When the display device 34 displays the saved data list 623 (Sb3), and the user U operates the operation device 35, the control device 51 (display control unit 514) displays the period setting screen 625 on the display device 34. The period setting screen 625 is used to set the saving period for each piece of recorded data R1. The saving period is the period during which the recorded data R is effectively saved in the storage device 52.

[0140] User U operates the operating device 35 to instruct the retention period for each piece of recorded data R1. For example, the retention period is specified by the duration from the setting time (e.g., one month) or the date and time of the end of the period. The control device 51 (information processing unit 512) stores the retention period specified by user U in association with the recorded data R1 in the storage device 52. In addition to setting the retention period, user U can also prevent the deletion of recorded data R1.

[0141] The control device 51 (display control unit 514) determines whether the current date and time are included in the storage period of each recorded data R1. The control device 51 deletes the recorded data R1 that has been stored for a period of time from among the multiple recorded data R1. In addition, it does not delete recorded data R1 that has been prohibited from being deleted.

[0142] In the fifth embodiment, the same effect as in the first embodiment is achieved. Furthermore, in the fifth embodiment, since the retention period for the recorded data R1 is set, the possibility of a large amount of recorded data R1 remaining unused in the storage device 52 for an extended period can be reduced. Therefore, the required storage capacity of the storage device 52 can be reduced.

[0143] F: Implementation Method 6

[0144] Figure 16 This is a schematic diagram of reference screen 61 of the sixth embodiment. Reference screen 61 of the sixth embodiment displays the period during which performer Ua made a performance error (hereinafter referred to as the "performance error period"). The performance evaluation unit 513 of the sixth embodiment determines the performance error period by comparing the performance represented by performance data D with the demonstration performance represented by reference data. For example, a period in which the number of differences between performer Ua's performance and the demonstration performance represented by reference data exceeds a predetermined threshold is determined as a performance error period.

[0145] The display control unit 514 displays the period image B5, representing the period of the performance error, together with the period image B1 in the performance area G1. Specifically, the performance error period is displayed based on a time axis τ shared with other object periods such as the performance period and the speaking period. Furthermore, in Figure 16The example shown illustrates the case where the period image B5 during a performance error is displayed in the performance area G1, but the period image B5 can also be displayed in an area G that is independent of the performance area G1.

[0146] In the sixth embodiment, the same effect as in the first embodiment is achieved. Furthermore, in the sixth embodiment, the periods of performance errors by performer Ua that differ from the demonstration performance are displayed based on a timeline τ shared with both the performance and speaking periods. Therefore, the user U can visually and intuitively grasp the periods during which performance errors occurred by performer Ua.

[0147] G: Implementation Method 7

[0148] In the aforementioned embodiments, for convenience, a scenario is envisioned where the performer Ua plays the keyboard instrument 10 and the instructor Ub speaks to provide guidance. However, scenarios are also envisioned, for example, where, in a music classroom, the instructor Ub, in addition to the performer Ua, also plays the keyboard instrument 10, or where, in addition to the instructor Ub, the performer Ua also speaks. The seventh embodiment is a method for displaying a reference screen 61 related to the performance and speech of both the performer Ua and the instructor Ub.

[0149] Figure 17 This is a schematic diagram of the image displayed on the display device 34 of the terminal device 30 that has logged into the control system 50. On the display device 34 of the terminal device 30, an authentication screen 671 is displayed for the instructor Ub to input authentication information (e.g., identification information and password). If the instructor Ub's authentication is successful, a response screen 672 is displayed, prompting whether there is input from other users U who share the recorded data R1. If sharing with other users U is confirmed, an authentication screen 673 is displayed for the performer Ua to input authentication information. By repeatedly switching between the displays illustrated above, multiple users U (performer Ua, instructor Ub) can each log in.

[0150] In addition, such as Figure 18 As illustrated, after the authentication screen 671 of the instructor Ub is displayed, an instruction screen 674 can be displayed to designate one or more performers Ua who share the recording data R1 with the instructor Ub, and a confirmation screen 675 can be displayed to confirm whether the designation of the performers Ua is appropriate.

[0151] The performance data D in the seventh embodiment represents the performance of the keyboard instrument 10, which includes the performance of the performer Ua and the performance of the instructor Ub. Additionally, the speech audio represented by the audio data A includes the speech audio of the performer Ua and the speech audio of the instructor Ub.

[0152] Furthermore, the method for distinguishing between the performance of performer Ua and the performance of instructor Ub is arbitrary. For example, either performer Ua or instructor Ub can be designated through operation of the operating device 15 of the keyboard instrument 10 or the operating device 35 of the terminal device 30. The control device 51 (information processing unit 512) distinguishes between the performance of performer Ua and the performance of instructor Ub based on the operation of the operating device 15 or the operating device 35.

[0153] Furthermore, the method for distinguishing the audio of the performer Ua's speech from the audio of the instructor Ub's speech is arbitrary. For example, either the performer Ua or the instructor Ub can be designated through operation of the operating device 15 of the keyboard instrument 10 or the operating device 35 of the terminal device 30. The control device 51 (information processing unit 512) distinguishes the audio of the performer Ua's speech from the audio of the instructor Ub based on the operation of the operating device 15 or the operating device 35. Alternatively, the audio of the performer Ua's speech from the audio of the instructor Ub can also be distinguished based on speaker identification of the audio data A.

[0154] Figure 19 This is a schematic diagram of reference screen 61 according to the seventh embodiment. Reference screen 61 of the seventh embodiment includes a performance area G1a, a performance area G1b, a speech area G3a, and a speech area G3b. Furthermore, other areas G such as the evaluation area G2 and the image area G4 are omitted for convenience. Figure 19 The middle part is omitted, but it can also be included in reference screen 61 in the same way as the aforementioned methods.

[0155] The performance area G1a is the area that displays the performance period related to the performance of performer Ua. Specifically, the performance area G1a displays a display item name N1 and a performance image B1. The display item name N1 contains the name of performer Ua, "UserA," and the name of the displayed object, "Performance." The performance image B1 is an image representing the performance period in which performer Ua is performing.

[0156] The performance area G1b is the area that displays the performance period related to the instructor Ub's performance. Specifically, the performance area G1b displays a display item name N1 and a performance image B1. The display item name N1 contains the instructor Ub's name "UserB" and the name of the displayed object "Performance". The performance image B1 is an image representing the performance period in which the instructor Ub's performance takes place.

[0157] As explained above, the multiple performance periods shown in reference screen 61 of the seventh embodiment include a performance period in which performer Ua performs and a performance period in which instructor Ub performs. Furthermore, performer Ua's performance is an example of a "first performance," and performer Ua's performance period is an example of a "first performance period." Additionally, instructor Ub's performance is an example of a "second performance," and instructor Ub's performance period is an example of a "second performance period."

[0158] In the seventh embodiment, the performance periods of performer Ua and instructor Ub are displayed on the display device 34 based on a common time axis τ. Therefore, the user U can visually and intuitively grasp the temporal relationship between the performances of performer Ua and instructor Ub.

[0159] The speech area G3a is the area that displays the speech period related to the performer Ua's speech. Specifically, the speech area G3a displays a display item name N3 and a period image B3. The display item name N3 contains the performer Ua's name "UserA" and the name of the displayed object "Audio". The period image B3 is an image representing the speech period in which the performer Ua's speech is present.

[0160] The speech area G3b is the area that displays the speech period related to the instructor Ub's speech. Specifically, the speech area G3b displays a display item name N3 and a period image B3. The display item name N3 contains the instructor Ub's name "UserB" and the name of the displayed object "Audio". The period image B3 is an image representing the speech period in which the instructor Ub's speech is present.

[0161] As explained above, the multiple speaking periods displayed on reference screen 61 in the seventh embodiment include a speaking period containing the audio of performer Ua's speech and a speaking period containing the audio of instructor Ub's speech. Furthermore, the audio of performer Ua's speech is an example of "first speaking audio," and the speaking period of performer Ua is an example of "first speaking period." Additionally, the audio of instructor Ub's speech is an example of "second speaking audio," and the speaking period of instructor Ub is an example of "second speaking period."

[0162] In the seventh embodiment, the speaking periods of the performer Ua and the speaking periods of the instructor Ub are displayed on the display device 34 based on a common time axis τ. Therefore, the user U can visually and intuitively grasp the temporal relationship between the speaking audio of the performer Ua and the speaking audio of the instructor Ub.

[0163] Furthermore, for example in a music classroom, sometimes one instructor Ub instructs multiple performers Ua. The performances and conversations of the multiple performers Ua (Ua1, Ua2) shared by instructor Ub can also be displayed on reference screen 61. For example, the performance periods of performers Ua1 and Ua2 are displayed on display device 34 based on a common timeline τ. Additionally, the conversation periods of performers Ua1 and Ua are displayed on display device 34 based on a common timeline τ. Furthermore, the performances and conversations related to multiple performers Ua and instructor Ub can also be displayed side-by-side on reference screen 61.

[0164] Furthermore, it is envisioned that the information processing system 100 includes multiple sets of keyboard instruments 10 and terminal devices 30. The aforementioned methods are also applicable in situations where multiple performers Ua or multiple instructors Ub each utilize the keyboard instruments 10 and terminal devices 30.

[0165] H: Implementation Method 8

[0166] As described above, the recorded data R1 is stored in the storage device 52 for each of the multiple users U (performer Ua and instructor Ub). The eighth embodiment is a method for comparing the performances of multiple users U.

[0167] Figure 20 This is a schematic diagram of the image displayed on the display device 34 of the terminal device 30 in the eighth embodiment. The control device 51 (display control unit 514) displays a selection screen 681 on the display device 34 based on the operation of the operation device 35 by the user U. The selection screen 681 is an image for selecting multiple users U (hereinafter referred to as "display subjects") as comparison objects. For example, the performer Ua or the instructor Ub is selected as the display subject.

[0168] If multiple display objects are selected, the control device 51 (display control unit 514) displays detailed information E for each of the multiple display objects in a comparative manner. Figure 20 In the process, for both the performer Ua and the instructor Ub, the performance image E1, partial image E3, and performance parameter conversion E4 are displayed on the display device 34.

[0169] In addition, Figure 20In this model, the performer Ua and the instructor Ub are conceived as the display objects. The performance image E1 of the performer Ua is superimposed on the performance image E1 of the instructor Ub. Therefore, the user U can easily compare the performance of the keyboard instrument 10 between the performer Ua and the instructor Ub. Furthermore, the performance parameter conversion E4 of the performer Ua is superimposed on the performance parameter conversion E4 of the instructor Ub. Therefore, the user U can easily compare the performance parameter conversion E4 between the performer Ua and the instructor Ub.

[0170] In the eighth embodiment, the same effect as in the first embodiment is achieved. Furthermore, in the eighth embodiment, since detailed information E is displayed in a comparative manner for multiple display objects, the user U is able to visually compare the performances between the display objects.

[0171] I: Implementation Method 9

[0172] Figure 21 This is a schematic diagram of the image displayed on the display device 34 of the terminal device 30 in the ninth embodiment. The control device 51 (display control unit 514) displays a selection screen 691 on the display device 34 based on the user U's operation of the operation device 35. Similar to the eighth embodiment, the selection screen 691 is an image for selecting multiple display objects.

[0173] If multiple subjects are selected, the control device 51 (display control unit 514) displays a scale image 692 on the display device 34. The scale image 692 is a pie chart showing the proportions of the performance and speaking periods relative to the duration of a single lesson in the music classroom for each of the multiple subjects. Specifically, the scale image 692, representing the proportions of the performance period for performer Ua (UserA / performance), the speaking period for performer Ua (UserA / audio), the performance period for instructor Ub (UserB / performance), and the speaking period for instructor Ub (UserA / audio), is displayed on the display device 34. By referring to the scale image 692, the user U can visually and intuitively grasp the composition of a single lesson.

[0174] Furthermore, the control device 51 (display control unit 514) takes advantage of the user U's operation of the operating device 35 to... Figure 22 Selection screen 693 is displayed on display device 34. Similar to the eighth embodiment, selection screen 693 is an image for selecting one display subject from multiple candidates.

[0175] If a subject is selected, the control device 51 (display control unit 514) displays a scale image 694 on the display device 34. The scale image 694 is a pie chart displaying the time ratio between the subject's performance and speaking periods. For example, the time ratio of instructor Ub's performance (UserB / performance) and the time ratio of instructor Ub's speaking (UserB / audio) are displayed on the display device 34. By referring to the scale image 694, the user U can visually and intuitively grasp the temporal relationship between the subject's performance and speaking periods. Therefore, the scale image 694 is used, for example, as effective data for evaluating the appropriateness of the instruction given by the subject, i.e., instructor Ub.

[0176] As understood from the above examples, the display control unit 514 of the ninth embodiment displays a scale image (692, 694) on the display device 34 that represents the time proportions related to multiple object periods, including the performance period and the speech period.

[0177] J: Variation Example

[0178] The following examples illustrate specific variations that can be added to the methods illustrated above. Two or more methods selected from the examples below may be combined, provided they do not contradict each other.

[0179] (1) Among the foregoing methods, the method in which the performer Ua plays the keyboard instrument 10 is exemplified, but the instrument played by the performer Ua is not limited to the keyboard instrument 10. That is, the present invention can be used to manage the performance of any kind of instrument.

[0180] (2) Among the aforementioned methods, multiple dynamic images obtained by capturing the performer Ua playing the keyboard instrument 10 from different directions are illustrated as part of the image E3. Figure 12 However, the content of some images E3 is not limited to the examples above. For example, as part of image E3, images of the hands of the player Ua playing the keyboard instrument 10, images of the instructor Ub in addition to the player Ua, or images of the feet of the player Ua operating the pedal 184 can be shown. Similarly, the editing animation E6 is not limited to images of the player Ua's hands.

[0181] (3) In the aforementioned methods, piano scroll images are exemplified as performance image E1 and performance parameter conversion E4, but the specific methods of performance image E1 and performance parameter conversion E4 are not limited to the examples above. For example, the operation of the performer Ua on pedal 184 can also be displayed through performance image E1 or performance parameter conversion E4. For example, performance image E1 representing the period during which the performer Ua operates on pedal 184 can also be displayed. In addition, in the method where the performance parameters include the amount of pedal 184 being pressed, the amount of pedal 184 being pressed by the performer Ua can also be displayed as performance parameter conversion E4.

[0182] (4) In the aforementioned methods, the reference data for the demonstration performance is, for example, data prepared in advance. However, the performance data D of the instructor Ub, who is performing with the performer Ua, can also be used as reference data. For example, the following operation mode can be set: if the performer Ua and the instructor Ub log in to the control system 50 through the terminal device 30, the performance data D output from the instructor Ub's keyboard instrument 10 will be used as reference data. The control device 51 (data acquisition unit 511) stores the performance data D representing the instructor Ub's performance as new reference data in the storage device 52.

[0183] Furthermore, when performer Ua and instructor Ub play the same keyboard instrument 10 separately or side-by-side, the data acquisition unit 511 can store the performance data D generated by instructor Ub's performance as reference data in the storage device 52, and the performance evaluation unit 513 can use this reference data to evaluate performer Ua's performance in real time. That is, the acquisition of reference data and the evaluation of performer Ua's performance can be performed in parallel and in real time with the performances of performer Ua and instructor Ub.

[0184] (5) In the foregoing embodiments, the information processing system 100 is shown to have a keyboard instrument 10, but the keyboard instrument 10 may also be omitted from the information processing system 100. For example, the keyboard instrument 10 may be installed in an acoustic space located remotely from the information processing system 100 (terminal device 30). The performance data D generated by the keyboard instrument 10 is transmitted to the information processing system 100, for example, via a communication network 200 such as the Internet. The operation of the terminal device 30 is the same as in the foregoing embodiments. As understood from the above description, the "information processing system" of the present invention may or may not have an instrument.

[0185] (6) Some or all of the functions of the control system 50 illustrated in the foregoing embodiments may also be carried on the terminal device 30.

[0186] (7) The functions of the control system 50 illustrated above are realized through the coordinated operation of one or more processors constituting the control device 51 and the program stored in the storage device 52, as described above. The program involved in this invention can be provided and installed in a computer in the form of a computer-readable recording medium. The recording medium is, for example, a non-transitory recording medium, preferably an optical recording medium (optical disc) such as a CD-ROM, and also includes any known form of recording medium such as a semiconductor recording medium or a magnetic recording medium. Furthermore, as a non-transitory recording medium, any recording medium other than a transient propagating signal is included, and volatile recording media may also be excluded. In addition, in a structure in which a transmission device transmits a program via a communication network, the storage device storing the program in the transmission device is equivalent to the aforementioned non-transitory recording medium.

[0187] K: Appendix

[0188] Based on the examples above, one can, for instance, grasp the following structure.

[0189] One aspect (Aspect 1) of the present invention relates to an information processing system comprising: a data acquisition unit that acquires performance data representing a performer's performance of an instrument and audio data representing speech audio related to the performer's performance; and a display control unit that displays a reference screen on a display device, the reference screen displaying, based on a common timeline, one or more performance periods containing the performance data and one or more speech periods containing the speech audio represented by the audio data. In the above-described aspect, the user (e.g., a performer or instructor) can visually and intuitively grasp the approximate timeline relationship between the performance periods containing the performer's performance and the speech periods containing the speech audio related to the performance.

[0190] "Performance data" is any form of data that represents a performer's playing of an instrument. For example, "performance data" can be exemplified by time-series data in MIDI format representing each note played by the performer, or audio data that records the sounds of the performer playing the instrument.

[0191] "Audio data" refers to any form of data representing the audio of a performer speaking in the venue where the performer is performing (speech audio). For example, examples of "audio data" include audio representing the performer's own audio or audio representing the audio of an instructor guiding the performer. Additionally, audio data of listeners listening to the performer's performance is also included in "audio data".

[0192] In addition to acquiring performance data representing the performance of a single performer, the data acquisition unit can also acquire multiple performance data representing the performances of different performers on the instruments. Furthermore, the source of the performance data acquired by the data acquisition unit is arbitrary. For example, the data acquisition unit can acquire performance data from a device directly connected to the information processing system, or it can acquire performance data from a remote device that communicates with the information processing system.

[0193] In addition to acquiring audio data representing speech audio from one location, the data acquisition unit can also acquire multiple audio data representing speech audio from different locations. Furthermore, the source from which the data acquisition unit acquires audio data is arbitrary. For example, the data acquisition unit can acquire audio data from a device directly connected to the information processing system, or it can acquire audio data from a remote device that communicates with the information processing system.

[0194] A “performance period” is a continuous series of periods during which a performer plays an instrument. On the other hand, a “speech period” is a continuous series of spoken audio periods. A “speech period” may, for example, include short silent periods within a speech period.

[0195] A “reference image” is an image of any form that represents the performance and speaking periods on a timeline. For example, as a “reference image,” an image showing the performance and speaking periods based on a common timeline is illustrated.

[0196] In a specific example of Method 1 (Method 2), an information processing unit is further included. This information processing unit uses the performance data and the audio data to generate period data that specifies the performance period and the speech period. The display control unit uses the period data to display the reference screen. In the above methods, period data that specifies the performance period and the speech period is generated, and the reference screen is displayed using the period data.

[0197] "Period data" refers to data that specifies the start and end points of each period on a timeline. For example, "period data" shows data specifying the start and end points for each performance period and speech period.

[0198] In a specific example of Method 1 or 2 (Method 3), the more than one performance period is a plurality of performance periods. The information processing system further includes: a selection receiving unit that accepts a selection of any one of the plurality of performance periods in the reference screen; and an information output unit that, when the selection receiving unit receives a selection of the performance period, outputs detailed information indicating the content of the performance within the performance period. In the above methods, it is possible to confirm the detailed content of the performance for the performance period desired by the user among the plurality of performance periods.

[0199] "Details" refers to any form of image or audio representing the content of a performance during the performance. For example, examples of "details" include performance images (such as piano scroll images) representing a string of notes during the performance, performance animations obtained by taking pictures of the performers during the performance, or audio representing these images.

[0200] In any specific example of methods 1 to 3 (method 4), the speech audio represented by the audio data includes a first speech audio of the performer and a second speech audio of the instructor guiding the performer, and the more than one speech period in the reference screen includes a first speech period containing the first speech audio and a second speech period containing the second speech audio. In the above methods, the first speech period containing the first speech audio of the performer and the second speech period containing the second speech audio of the instructor guiding the performer are displayed. Therefore, the temporal relationship between the first speech audio and the second speech audio can be grasped visually and intuitively.

[0201] In any specific example of methods 1 to 4 (method 5), the performance data represents a performance that includes a first performance by a first performer and a second performance by a second performer different from the first performer. The more than one performance period in the reference screen includes a first performance period in which the first performance exists and a second performance period in which the second performance exists. In the above methods, the first performance period in which the first performance by the first performer exists and the second performance period in which the second performance by the second performer exists are displayed. Therefore, the temporal relationship between the first performance and the second performance can be grasped visually and intuitively.

[0202] The first and second performers can be combinations of two different performers, or combinations of performers and instructors. In addition, the first performance by the first performer and the second performance by the second performer can be performed in parallel, or they can be recorded sequentially at different times.

[0203] In any specific example of methods 1 to 5 (method 6), a performance evaluation unit is further included. This unit compares the performance represented by the performance data with the demonstration performance represented by the reference data. The reference screen, based on a common timeline, displays one or more demonstration periods corresponding to performances in the performance represented by the performance data that are similar to the demonstration performance, as well as the one or more performance periods and the one or more speaking periods. In the above methods, based on a timeline shared with the performance and speaking periods, the demonstration periods corresponding to performances in the performer's performance that are similar to the demonstration performance are displayed. Therefore, the user can visually and intuitively grasp the appropriate performance periods in the performer's performance.

[0204] In any specific example of methods 1 to 5 (method 7), a performance evaluation unit is further included. This performance evaluation unit compares the performance represented by the performance data with the demonstration performance represented by reference data. The reference screen, based on a common timeline, displays one or more performance error periods corresponding to performances different from the demonstration performance in the performance represented by the performance data, as well as the one or more performance periods and the one or more speaking periods. In the above methods, based on a timeline shared with the performance and speaking periods, the performance error periods corresponding to performances different from the demonstration performance in the performer's performance are displayed. Therefore, the user can visually and intuitively grasp the periods during which performance errors occurred in the performer's performance.

[0205] In any specific example of methods 1 to 7 (method 8), the display control unit also displays a proportion image on the display device, which represents the time proportions related to multiple periods including the more than one performance period and the more than one speech period. In the above methods, the user can visually and intuitively grasp the time proportions of each period, including the more than one performance period and the more than one speech period. The time proportions of each period are useful, for example, for assessing the appropriateness of a performer's practice or an instructor's guidance.

[0206] One aspect (aspect 9) of the present invention relates to an information processing method that obtains performance data representing a performer's performance on an instrument and audio data representing speech audio related to the performer's performance, and displays a reference screen on a display device, the reference screen being based on a common timeline and displaying one or more performance periods during which the performance data represents the performance and one or more speech periods during which the audio data represents the speech audio.

[0207] One aspect of the present invention (aspect 10) involves a program that causes a computer system to function as the following functional units: a data acquisition unit that acquires performance data representing a performer's performance on an instrument and audio data representing speech audio related to the performer's performance; and a display control unit that displays a reference screen on a display device, the reference screen displaying, based on a common timeline, one or more performance periods in which the performance data represents the performance and one or more speech periods in which the audio data represents the speech audio.

[0208] Explanation of the label

[0209] 100…Information processing system, 200…Communication network, 10…Keyboard instrument, 11…Control device, 12…Storage device, 13…Communication device, 14…Display device, 15…Operating device, 16…Sound source device, 17…Sound playback device, 18…Playing mechanism, 181…Keyboard, 182…Keys, 183…Sound production mechanism, 184…Pedal, 185…Detection device, 20…Recording system, 21…Shooting device, 22…Pickup device, 30 (30, 30b, 30c)…Terminal device, 31 …control device, 32…storage device, 33…communication device, 34…display device, 35…operation device, 36…sound playback device, 37…sound pickup device, 40…measurement system, 41…biosensor, 42…center of gravity sensor, 50…control system, 51…control device, 52…storage device, 53…communication device, 511…data acquisition unit, 512…information processing unit, 513…performance evaluation unit, 514…display control unit, 515…selection and reception unit, 516…information output unit.

Claims

1. An information processing system, comprising: The data acquisition unit acquires performance data representing a performer's playing of an instrument and audio data representing speech audio related to the performer's playing; and The display control unit displays a reference screen on the display device. This reference screen is based on a common timeline and displays one or more performance periods where the performance data represents a performance and one or more speech periods where the audio data represents a speech audio.

2. The information processing system according to claim 1, wherein, It also includes an information processing unit that uses the performance data and the audio data to generate period data that specifies the performance period and the speech period. The display control unit uses the period data to display the reference screen.

3. The information processing system according to claim 1, wherein, The term "one or more performance periods" refers to multiple performance periods. The information processing system also has: The selection receiving unit accepts the selection of any one of the plurality of performance periods in the reference screen; as well as The information output unit, when the selection receiving unit receives a selection for the performance period, outputs detailed information indicating the content of the performance during the performance period.

4. The information processing system according to claim 1, wherein, The audio data represents the speech audio, which includes the first speech audio of the performer and the second speech audio of the instructor who guided the performer. The more than one speaking period in the reference screen includes a first speaking period containing the first speaking audio and a second speaking period containing the second speaking audio.

5. The information processing system according to claim 1, wherein, The performance data represents a performance that includes a first performance by a first performer and a second performance by a second performer who is different from the first performer. The more than one performance period in the reference frame includes a first performance period in which the first performance exists and a second performance period in which the second performance exists.

6. The information processing system according to claim 1, wherein, It also includes a performance evaluation department that compares the performance represented by the performance data with the demonstration performance represented by reference data. The reference screen displays one or more demonstration periods, one or more performance periods, and one or more speaking periods corresponding to the performances that are similar to the demonstration performances represented by the performance data, based on a common timeline.

7. The information processing system according to claim 1, wherein, It also includes a performance evaluation department that compares the performance represented by the performance data with the demonstration performance represented by reference data. The reference screen displays one or more performance errors corresponding to performances different from the demonstration performance, as well as the one or more performance periods and the one or more speaking periods, based on a common timeline.

8. The information processing system according to claim 1, wherein, The display control unit also displays a scale image on the display device, which represents the time proportions related to multiple periods including the one or more performance periods and the one or more speech periods.

9. An information processing method implemented by a computer system. Acquire performance data representing the performer's playing of the instrument and audio data representing speech audio related to the performer's playing. A reference screen is displayed on the display device. The reference screen is based on a common timeline and shows one or more performance periods where the performance data represents a performance and one or more speech periods where the audio data represents speech audio.

10. A program that causes a computer system to function as the following functional units: The data acquisition unit acquires performance data representing a performer's playing of an instrument and audio data representing speech audio related to the performer's playing; and The display control unit displays a reference screen on the display device. This reference screen is based on a common timeline and displays one or more performance periods where the performance data represents a performance and one or more speech periods where the audio data represents a speech audio.

Images