Program, information processing device, electronic musical instrument and display method

Abstract

To enable a user to easily confirm a difference of performance expression between model performance and user performance.SOLUTION: A control part of an information processing device derives a first smoothing processing value obtained by smoothing an index value on performance information on model performance and a second smoothing processing value obtained by smoothing an index value on performance information of user performance and displays a difference between the first smoothing processing value and the second smoothing processing value on a display part.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] The present invention relates to a program, an information processing device, an electronic musical instrument, and a display method. [Background technology]

[0002] A technique has been disclosed in which performance information (pitch, velocity, etc.) of a model performance and performance information (user performance) are plotted in association with each other for each beat and displayed as a line graph (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Utility Model Application Publication No. 4-35172 Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, performance expression should be considered as a continuous change as the song progresses. However, the technology described in Patent Document 1 displays a graph by connecting the discrete performance information data of the model performance and the user's performance with straight lines, making it difficult to grasp a continuous flow and for the user to easily confirm the difference in performance expression between the model performance and the user's performance.

[0005] An object of the present invention is to enable a user to easily confirm the difference in performance expression between a model performance and a user's performance. [Means for solving the problem]

[0006] In order to solve the above problems, the program of the present invention comprises: Computer, The computer is caused to function as a control unit that derives a first smoothed value obtained by smoothing an index value relating to performance information of a model performance and a second smoothed value obtained by smoothing an index value relating to performance information of a user performance, and causes a display unit to display the difference between the first smoothed value and the second smoothed value. Alternatively, the computer is caused to function as a control unit that derives a difference index value indicating a difference in expression between a model performance and a user performance based on performance information of a model performance and performance information of a user performance, smoothes a time change of the derived index value, and causes a display unit to visualize and display the time change of the smoothed index value. Effect of the Invention

[0007] According to the present invention, a user can easily confirm the difference in performance expression between a model performance and a user's performance. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an example of the overall configuration of a performance expression visualization system according to an embodiment of the present invention. [Diagram 2] 4 is a flowchart showing the flow of a musical performance assistance process executed by the control unit of FIG. 1. [Diagram 3] 4 is a flowchart showing the flow of a user performance input process executed by the control unit in FIG. 1. [Figure 4] 3 is a flowchart showing the process of reproducing and recording started in step S5 of FIG. 2. [Diagram 5] 3 is a flowchart showing the flow of the visualization process executed in step S7 of FIG. 2. [Figure 6] 11A and 11B are diagrams for explaining the effect of a damper pedal. [Figure 7] 6 is a flowchart showing a process of acquiring a substantial note length in response to a damper pedal operation, which is executed in step S701 of FIG. 5. [Figure 8] 7 is a flowchart showing the process of smoothing the time change of the index value, which is executed in step S711 of FIG. 5. [Figure 9]13 is a graph showing changes over time in the difference in length between a model performance and a user's performance, the difference in timing between the model performance and the user's performance, the strength of the model performance, and the strength of the user's performance. [Figure 10] 11 is a graph showing changes over time in the tempo of a model performance, the tempo of a user's performance, the strength of the model performance, and the strength of the user's performance. [Figure 11] 13 is a graph showing a change over time in an index value indicating a difference in effective note length in response to damper pedal operation between a model performance and a user performance. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments of the present invention will be described with reference to the drawings. However, the embodiments described below are subject to various technically preferable limitations for implementing the present invention. Therefore, the technical scope of the present invention is not limited to the following embodiments and illustrated examples.

[0010] [Configuration of performance expression visualization system 100] FIG. 1 is a diagram showing an example of the overall configuration of a performance expression visualization system 100 according to the present invention. 1, the performance expression visualization system 100 is configured to include an information processing device 1 and an electronic musical instrument 2. The information processing device 1 and the electronic musical instrument 2 are communicatively connected to each other wirelessly or via a wire so as to be able to transmit and receive data.

[0011] [Configuration of information processing device 1] The information processing device 1 is a device capable of visualizing and displaying the difference in performance expression between a model performance and a performance (user performance) performed by a user (performer) on an electronic musical instrument 2. The information processing device 1 may be, for example, a tablet PC (Personal Computer), a notebook PC, a smartphone, or the like.

[0012] As shown in FIG. 1, the information processing device 1 includes a control unit 11, a storage unit 12, an input unit 13, a display unit 14, a communication unit 15, an output unit 16, etc., and each unit is connected by a bus 17.

[0013] The control unit 11 is a computer that includes at least one processor such as a CPU (Central Processing Unit) and a RAM (Random Access Memory), and controls each part of the information processing device 1. Specifically, the CPU of the control unit 11 reads out a specified program from among the system programs and various application programs stored in the storage unit 12, expands it in the RAM, and executes various processes in cooperation with the expanded program. For example, when an instruction is given to start a performance assistance application based on a performance assistance app 121 as a performance assistance application program stored in the storage unit 12 by operating the input unit 13, the CPU of the control unit 11 executes a performance assistance process (see FIG. 2) described later. A recording buffer, a note-off buffer, an index value storage area, and the like, which will be described later in the performance assistance process, are formed in the RAM of the control unit 11.

[0014] The storage unit 12 is composed of a non-volatile semiconductor memory, a HDD (Hard Disk Drive), etc. The storage unit 12 stores the system program of the information processing device 1, various application programs, and data required for executing the programs. The storage unit 12 is not limited to being built into the information processing device 1, and may include an external recording medium that is detachable from the information processing device 1, such as an external HDD, a USB memory, etc. In this embodiment, the storage unit 12 stores a performance assistance application 121 for the control unit 11 to execute the performance assistance process shown in Fig. 2. The performance assistance application 121 refers to a program code or source code of a performance assistance application for executing the performance assistance process. The storage unit 12 also stores performance information (e.g., Musical Instrument Digital Interface (MIDI) data) of model performance of each piece of music that the user can play using the performance support application 121. The performance information of the model performance (hereinafter, referred to as model performance information) includes a Note ON event (Note ON information) including information on the timing (timekeeping timing such as time), pitch, velocity value (strength), and length (duration) of each Note ON (key pressing operation = performance operation), a Pedal ON event (Pedal ON information) including information on the timing of Pedal ON (pedal pressing operation), and a Pedal OFF event (Pedal OFF information) including information on the timing of Pedal OFF (pedal release operation). The timing is expressed as the elapsed time (in tick units) from the start of the performance. The length is also expressed in tick units. Tick is a unit of time length in a piece of music. For example, if one beat is 480 (Ticks), one measure of 4 / 4 time is 1920 (Ticks). The model performance information also includes tempo information and beat information. The tempo information includes a bpm value that indicates the tempo, and information on the timing (in ticks) at which performance at that tempo will begin. The beat information includes information on the beat (e.g., 4 / 4 beat), and information on the timing (in ticks) at which performance at that beat will begin. In the present application, a performance operation refers to an operation for generating musical instrument sounds, that is, a key pressing operation.

[0015] The input unit 13 is composed of push button switches, a touch panel attached to the display unit 14, etc. The input unit 13 detects the operation of the push button switches or the touch operation on the screen by the user, and outputs an operation signal to the control unit 11.

[0016] The display unit 14 is configured with an LCD (Liquid Crystal Display), an EL (Electro Luminescence) display, or the like, and performs various displays according to display information instructed by the control unit 11.

[0017] The communication unit 15 includes a wired unit or a wireless unit for communicating with external devices such as the electronic musical instrument 2, and transmits and receives data to and from the external devices. In this embodiment, the communication unit 15 transmits and receives data to and from the electronic musical instrument 2 via a communication interface (in this embodiment, a MIDI interface) not shown.

[0018] The output unit 16 includes a sound source, a D / A converter, an amplifier, a speaker, etc., and outputs sound based on instructions from the control unit 11.

[0019] [Electronic Instruments 2] The electronic musical instrument 2 is, for example, a keyboard instrument. As shown in FIG. 1, the electronic musical instrument 2 is configured with a control unit 21, a memory unit 22, a keyboard 23, an input unit 24, a display unit 25, a communication unit 26, an output unit 27, a pedal 28, etc., and each unit is connected by a bus 29.

[0020] The control unit 21 is a computer that includes at least one processor such as a CPU and a DSP (Digital Signal Processor), a RAM, etc., and controls each part of the electronic musical instrument 2. Specifically, the CPU of the control unit 21 reads out a specified program from among the system programs and various programs stored in the storage unit 22, loads it into the RAM, and executes various processes in cooperation with the loaded program. For example, the control unit 21 causes the output unit 27 to output (sound) musical instrument sounds based on the operation of the keyboard 23 and pedals .

[0021] Furthermore, when the control unit 21 detects an operation of the keyboard 23 or the pedal 28 while connected to the information processing device 1, the control unit 21 outputs performance information of the user performance indicating the detected operation (e.g., MIDI data) to the information processing device 1 via the communication unit 26. The performance information of the user performance (hereinafter referred to as user performance information) includes a note-on event (note-on information), a note-off event (note-off information), a pedal-on event (pedal-on information), and a pedal-off event (pedal-off information). For example, when the control unit 21 detects a key pressing operation (performance operation) on the keyboard 23, it outputs a Note ON event indicating the timing of the key pressing, the pitch of the pressed key, and a velocity value to the information processing device 1 via the communication unit 26. When the control unit 21 detects a key release operation on the keyboard 23, it outputs a Note OFF event indicating the timing of the key release and the pitch of the released key to the information processing device 1 via the communication unit 26. When the control unit 21 detects a pedal depression (pedal ON) on the pedal 28, it outputs a pedal ON event indicating the timing of the pedal ON to the information processing device 1 via the communication unit 26. When the control unit 21 detects a pedal release (pedal OFF) on the pedal 28, it outputs a pedal OFF event indicating the timing of the pedal OFF to the information processing device 1 via the communication unit 26. The timing is expressed as the elapsed time (in ticks) from the start of performance, similar to the model performance information.

[0022] The storage unit 22 is composed of a non-volatile semiconductor memory, a HDD (Hard Disk Drive), etc. The storage unit 22 stores the system program of the electronic musical instrument 2, various programs, and data required for executing the programs. The storage unit 22 is not limited to being built into the electronic musical instrument 2, and may include an external recording medium that is detachable from the electronic musical instrument 2, such as an external HDD, a USB memory, etc.

[0023] The keyboard 23 includes a plurality of keys (performance operators) and a detection unit that detects pressed / released keys, and outputs information (events) indicating the pitch, velocity and timing of the keys pressed / released by the user to the control unit 21.

[0024] The input unit 24 is composed of push button switches, a touch panel attached to the display unit 25, and the like. The input unit 24 detects the operation of the push button switches or the touch operation on the screen by the user, and outputs an operation signal to the control unit 21.

[0025] The display unit 25 is configured with an LCD (Liquid Crystal Display) or the like, and performs various displays according to display information instructed by the control unit 21.

[0026] The communication unit 26 includes a wired unit or a wireless unit for communicating with external devices such as the information processing device 1, and transmits and receives data to and from the external devices. In this embodiment, the communication unit 26 transmits and receives data to and from the information processing device 1 via a communication interface (in this embodiment, a MIDI interface) not shown.

[0027] The output unit 27 includes a sound source unit, a D / A converter, an amplifier, a speaker, etc. In accordance with an instruction from the control unit 21, the output unit 27 reads out waveform data stored in advance in a waveform ROM included in the sound source unit or generates waveform data, and outputs musical instrument sounds based on the waveform data from the speaker via the D / A converter and the amplifier.

[0028] The pedal 28 includes a damper pedal and a detection unit that detects the ON / OFF state of the damper pedal, and outputs ON / OFF information of the damper pedal to the control unit 21. Note that, although the present embodiment will be described taking as an example a case in which the pedal 28 is a damper pedal, the pedal 28 may also be configured to further include a soft pedal or a sostenuto pedal.

[0029] [Operation of the performance expression visualization system 100] Next, the operation of the performance expression visualization system 100 of this embodiment will be described. In the information processing device 1, when the input unit 13 instructs the start of the performance assistance application, the control unit 11 starts the performance assistance application based on the performance assistance app 121, and establishes a communication connection with the electronic musical instrument 2 through the communication unit 15. When the communication connection with the electronic musical instrument 2 is established, the control unit 11 displays an operation screen (not shown) on the display unit 14, which displays a selection field for a song to be played back or recorded, a playback / recording instruction button, a playback / recording end button, a visualization instruction button, a visualization end button, a close button (a button for instructing the end of the performance assistance application), and the like, and starts the performance assistance process. In this embodiment, playing and recording refers to recording a user performance performed using the model performance as a model while playing back the model performance. Visualization refers to graphing (displaying) the difference between the expression (performance expression) of the model performance and the user performance, i.e., the difference between the index value related to the performance information of the model performance and the index value related to the performance information of the user performance, on the display unit 14 or the display unit 25 of the electronic musical instrument 2, based on the model performance information stored in the storage unit 12 and the user performance information which is a recording of the user performance.

[0030] In this embodiment, the "timing," "length," "effective note length according to damper pedal operation," "strength," and "tempo" of performance operations are used as elements of performance expression, and the differences in these elements between the model performance and the user's performance are graphed. For "timing," "length," and "effective note length according to damper pedal operation," index values ​​(difference index values) indicating the differences in these values ​​between the model performance and the user's performance are calculated (derived), and the changes over time are graphed. For "strength" and "tempo," "strength" and "tempo" are obtained for the model performance and the user's performance, respectively, as index values ​​related to expression, and the changes over time of the respective index values ​​for the model performance and the user's performance are graphed and overlaid.

[0031] 2 is a flowchart showing the flow of the musical performance assistance process executed by the control unit 11 in cooperation with the musical performance assistance application 121. The musical performance assistance process will be described below with reference to FIG.

[0032] First, the control unit 11 performs initialization required for the operation of the musical performance assistance process (step S1). For example, the recording flag, which indicates whether recording is in progress, is turned OFF, and other variables used in the performance assistance process are initialized.

[0033] Next, the control unit 11 starts a user performance input process as a separate thread (step S2).

[0034] 3 is a flow chart showing the flow of the user performance input process after step S2. The user performance input process is executed by the control unit 11 and the performance assistance application 121 in cooperation with each other.

[0035] In the user performance input process, first, the control unit 11 determines whether or not user performance information has been input (received) from the electronic musical instrument 2 by the communication unit 15 (step S201). When it is determined by the communication section 15 that the user performance information has not been input from the electronic musical instrument 2 (step S201; NO), the control section 11 returns to step S201. In the electronic musical instrument 2, the output unit 27 produces musical instrument sounds in response to a user's performance (depression of the keys on the keyboard 23 or depression of the pedals 28).

[0036] When it is determined that user performance information has been input from the electronic musical instrument 2 (step S201; YES), the control unit 11 refers to the recording in progress flag and determines whether or not recording is in progress (step S202). The control unit 11 determines that recording is in progress when the recording flag is ON, and determines that recording is not in progress when the recording flag is OFF. Note that the recording flag is switched ON / OFF during the playback / recording process, which will be described later.

[0037] When it is determined that recording is in progress (step S202; YES), the control unit 11 stores the input user performance information in a recording buffer (step S203), and the process returns to step S201. The control unit 11 stores the input user performance information in the recording buffer in association with the value stored in the variable lNowTick. lNowTick is a variable managed by the playback and recording process described later, and stores a value that represents the elapsed time in tick units from the start of the playback and recording process, which is the start of the model performance and the user performance.

[0038] When it is determined that recording is not in progress (step S202; NO), the control unit 11 returns to step S201.

[0039] The control unit 11 repeatedly executes the processes of steps S201 to S203 until the musical performance assistance process ends.

[0040] Returning to FIG. 2, when the user performance input process is started, the control unit 11 judges whether or not an instruction to end the performance assistance application has been given (step S3). The control unit 11 determines whether or not an instruction to end the musical performance assistance application has been issued based on whether or not the input unit 13 has detected pressing of the close button.

[0041] When it is determined that the end of the musical performance assistance application has not been instructed (step S3; NO), the control unit 11 determines whether or not a playback / recording command has been issued (step S4). The control unit 11 determines that playback and recording has been instructed, for example, when the input unit 13 detects pressing of a playback and recording instruction button. Note that a default song is set as the target song for playback and recording, but the user can select a song from a selection field for the target song for playback and recording.

[0042] When it is determined that playback / recording has been instructed (step S4; YES), the control unit 11 executes playback / recording processing (step S5).

[0043] 4 is a flow chart showing the flow of the playback and recording process, which is executed by the control unit 11 and the performance assistance application 121 in cooperation with each other.

[0044] In the playback / recording process, first, the control unit 11 initializes variables necessary for the playback / recording process (step S501). For example, the control unit 11 sets the initial tempo information (bpm value) of the model performance information to the variable Tempo as an initial value, and stores the initial beat information (e.g., 4 / 4 beat) of the model performance information to the variable Beat as an initial value.

[0045] Next, the control unit 11 clears the recording buffer and the note-off buffer (step S502). The recording buffer is a buffer for recording the user's performance in the user performance input process. In other words, it is a buffer for storing user performance information. The note-off buffer is a buffer for temporarily storing note-off information in association with the note pitch of the model performance that was subjected to the note-on process in step S509 and the note-off timing.

[0046] Next, the control unit 11 sets the recording in progress flag to ON (step S503).

[0047] Next, the control unit 11 judges whether or not to end the playback / recording process (step S504). For example, when the reproduction of the target song to be reproduced and recorded is completed, or when the input unit 13 detects that the reproduction / recording end button has been pressed, the control unit 11 determines that the reproduction / recording process is to be terminated.

[0048] If it is determined that the playback / recording process is not to be ended (step S504; NO), a tick calculation is performed (step S505). In step S505, the control unit 11 converts the elapsed time (time measurement timing) from the start of the current playback / recording process into tick units based on the tempo information (current bpm value of the model performance) stored in the variable Tempo, and stores the converted value in the variable lNowTick.

[0049] Next, the control unit 11 executes tempo processing (step S506). The tempo process is a process that determines whether or not there has been a change in tempo based on the model performance information, and if there has been a change, stores (overwrites) the changed tempo in the variable Tempo.

[0050] Next, the control unit 11 executes time signature processing (step S507). The time signature process is a process of determining whether or not there is a change in the time signature based on the model performance information, and if there is a change, storing (overwriting) the changed time signature in the variable Beat.

[0051] Next, the control unit 11 executes a note-off process (step S508). In the Note OFF process, the control unit 11 refers to the Note OFF buffer, and when it determines that the Note OFF timing of any of the Note OFF information stored in the Note OFF buffer has been reached, it instructs the electronic musical instrument 2 to cause the output unit 27 to stop the sound of the pitch specified by that Note OFF information, and deletes that Note OFF information from the Note OFF buffer. Note that if the pedal 28 is currently ON, even if it is the Note OFF timing, it will not instruct the output unit 27 to stop outputting the sound or delete the sound from the Note OFF buffer until the pedal 28 is turned OFF.

[0052] Next, the control unit 11 executes a note-on process (step S509), and returns to step S504. In the Note ON process, the control unit 11 judges whether or not the Note ON timing has been reached based on the model performance information, and if it judges that the Note ON timing has been reached, it instructs the output unit 27 to cause the electronic musical instrument 2 to produce an instrument sound of the pitch specified in the Note ON event that has become the Note ON timing, at a specified velocity value (strength).The control unit 11 also determines the Note OFF timing based on the length specified in the Note ON event, and stores the Note OFF information including the Note OFF timing and the pitch in the Note OFF buffer.

[0053] In step S504, when it is determined that the playback / recording process is to be ended (step S504; YES), the control unit 11 sets the recording in progress flag to OFF (step S510). Then, the control unit 11 converts the format of the user performance information stored in the recording buffer to match the model performance information (step S511), and the process returns to step S3 in FIG.

[0054] As described above, the user performance information includes a pair of Note ON and Note OFF events, and the timing of pressing / releasing a key is specified by a pair of Note ON and Note OFF events. Meanwhile, in the model performance information, the timing of pressing / releasing a key is described by the timing and length (note length) of the Note ON. Therefore, in step S511, the Note ON event of the user performance information is converted into a Note ON timing, a pitch, a velocity value, and a length. That is, the control unit 11 calculates the length of the Note ON (in tick units) based on the Note ON timing specified by the Note ON event in the user performance information and the Note OFF timing specified by the corresponding Note OFF event, converts the Note ON event into information including the length, and deletes the Note OFF event.

[0055] Returning to FIG. 2, in step S4, if it is determined that playback / recording has not been instructed (step S4; NO), the control unit 11 determines whether visualization has been instructed (step S6). When the control unit 11 detects that the visualization instruction button has been pressed by the input unit 13, it determines that visualization has been instructed.

[0056] When it is determined that visualization has not been instructed (step S6; NO), the control unit 11 returns to step S3. When it is determined that visualization has been instructed (step S6; YES), the control unit 11 executes visualization processing (step S7).

[0057] 5 is a flowchart showing the flow of the visualization process, which is executed by the control unit 11 and the performance assistance application 121 in cooperation with each other.

[0058] In the visualization process, first, the control unit 11 executes a process of acquiring a substantial note length corresponding to the damper pedal operation (step S701).

[0059] The damper pedal has the effect of extending the duration of a note that is being sounded (or is being sounded). There are two conditions for a note to have the damper pedal effect. One is that the pedal ON occurs before the note ON, and the next pedal OFF occurs after the note OFF (outside the note). If this condition is met, the difference between the note ON timing and the pedal OFF timing becomes the effective note length (the time the note is sounded) according to the damper pedal operation. In this case, it becomes longer than the note length from note ON to note OFF. For example, as shown in the upper part of Fig. 6(a), if the pedal turns ON at timing A1 before the note ON timing A2 and turns OFF at timing A4 after the note OFF timing A3, the note length from note ON to note OFF without considering the damper pedal operation is A2 to A3, but if the damper pedal operation is taken into consideration, the effective note length according to the damper pedal operation is A2 to A4. On the other hand, as shown in the lower part of Fig. 6(a), if the pedal turns ON at timing A1 before the note ON timing A2 and turns OFF at timing A4 before the note OFF timing A5, both the note length from note ON to note OFF without considering the damper pedal operation and the effective note length according to the damper pedal operation taking the damper pedal operation into consideration are A2 to A5.

[0060] The other is that the pedal ON occurs after the note ON, and the next pedal OFF occurs after the note OFF (outside the note). In this case, as in the first case, the difference between the note ON timing and the pedal OFF timing is the effective note length according to the damper pedal operation. For example, as shown in the upper part of Fig. 6(b), if the pedal turns ON at timing B2 after note ON timing B1 and turns OFF at timing B4 after note OFF timing B3, the note length from note ON to note OFF without considering damper pedal operation is B1 to B3, but if damper pedal operation is taken into consideration, the effective note length according to damper pedal operation is B1 to B4. On the other hand, as shown in the lower part of Fig. 6(b), if the pedal turns ON at timing B2 after note ON timing B1 and turns OFF at timing B4 before note OFF timing B5, both the note length from note ON to note OFF and the effective note length according to damper pedal operation are the same, B1 to B5. In the process of acquiring the substantial note length corresponding to the damper pedal operation, the substantial note length corresponding to the damper pedal operation is acquired based on the above conditions.

[0061] 7 is a flowchart showing the flow of the process of acquiring the substantial note length in response to the damper pedal operation. The process of acquiring the substantial note length in response to the damper pedal operation is executed by the control unit 11 and the performance assistance application 121 in cooperation with each other. The control unit 11 executes the process shown in Fig. 7 for each of the model performance information and the user performance information. If the model performance information has already been processed, it is not necessary to execute the process again.

[0062] In the process shown in FIG. 7, first, the control unit 11 initializes the pointer me (step T101). The pointer me is a variable for storing information (here, a number from the beginning) indicating an event to be processed among the events included in the performance information to be processed (model performance information or user performance information). In step T101, the pointer me is set to 0, which is information indicating the beginning event.

[0063] Next, the control unit 11 initializes a variable pedal_state to false (step T102). The variable pedal_state is set to true when the pedal 28 is ON, and is set to false when the pedal 28 is OFF.

[0064] Next, the control unit 11 judges whether or not the loop process of steps T103 to T127 has been completed up to the last event of the performance information (step T103). If the pointer me contains a number greater than the number of the last event in the performance information, the control unit 11 determines that the loop process from steps T103 to T127 has been completed up to the last event in the performance information.

[0065] If it is determined that the loop process of steps T103 to T127 has not been completed up to the last event of the performance information (step T103; NO), the control unit 11 determines whether the event indicated by the pointer me is a note-on event (step T104).

[0066] When it is determined that the event indicated by the pointer me is not a note-on event (step T104; NO), the control section 11 determines whether or not the event indicated by the pointer me is a pedal-on event (step T105). When it is determined that the event indicated by the pointer me is a pedal-ON event (step T105; YES), the control unit 11 sets the variable pedal_state to true (step T106), and the process proceeds to step T127. When it is determined that the event indicated by the pointer me is not a pedal-on event (step T105; NO), the control unit 11 sets the variable pedal_state to false (step T107), and proceeds to step T127.

[0067] In step T104, if it is determined that the event indicated by the pointer me is a note ON event (step T104; YES), the control unit 11 determines whether or not the damper pedal of the pedal 28 was turned ON before the note ON event indicated by the pointer me (step T108). If pedal_state=true, the control unit 11 determines that the damper pedal of pedal 28 was turned on before the note ON event indicated by the pointer me, and if pedal_state=false, it determines that the damper pedal of pedal 28 was not turned on before the note ON event indicated by the pointer me.

[0068] When it is determined that the damper pedal of the pedals 28 has been turned ON before the note ON event indicated by the pointer me (step T108; YES), the control unit 11 sets the information of the pointer me to the pointer next (step T109).

[0069] Next, the control unit 11 judges whether or not the loop process from T110 to T115 has been completed for all events following the note-on event indicated by the pointer me (step T110). If the pointer "next" contains a number greater than the number of the last event in the performance information, it is determined that the loop process from T110 to T115 has been completed for all events following the note-on event indicated by the pointer "me."

[0070] If it is determined that the loop processing from T110 to T115 has not been completed for all events after the note ON event indicated by pointer me (step T110; NO), the control unit 11 determines whether the event indicated by pointer next is a pedal OFF event and is outside the note of the note ON event indicated by pointer me (step T111). When the event pointed to by the pointer next is outside the note of the note ON event pointed to by the pointer me, it means that the event pointed to by the pointer next is an event after the note OFF of the note ON event pointed to by the pointer me.When the event pointed to by the pointer next is within the note of the note ON event pointed to by the pointer me, it means that the event pointed to by the pointer next is an event before the note OFF of the note ON event pointed to by the pointer me.

[0071] If it is determined that the event indicated by the pointer next is a pedal OFF event and is outside the note of the note ON event indicated by the pointer me (step T111; YES), the control unit 11 calculates the difference between the timing of the note ON event indicated by the pointer next (next->lTime) and the timing of the note ON event indicated by the pointer me (me->lTime), adds the calculation result to the information of the note ON event indicated by the pointer me as the effective note length (me->pedal_gate) corresponding to the damper pedal operation of the pedal 28 of the note ON event indicated by the pointer me (step T112), and proceeds to step T127.

[0072] If it is determined that the event indicated by pointer next is a pedal OFF event and is within the note of the note ON event indicated by pointer me (step T111; NO, step T113; YES), the control unit 11 adds the note length (me->lGate) of the note ON event indicated by pointer me to the information of the note ON event indicated by pointer me as the actual note length (me->pedal_gate) corresponding to the operation of pedal 28 of the note ON event indicated by pointer me (step T114), and proceeds to step T127. When it is determined that the event indicated by the pointer "next" is not a pedal-off event (step T111; NO, step T113; NO), the control unit 11 advances the number of the pointer "next" to the next number (step T115) and returns to step T110.

[0073] The control unit 11 repeatedly executes the loop process of steps T110 to T115 until it determines that the loop process of T110 to T115 has been completed for all events subsequent to the note-on event indicated by the pointer me. In step T110, when it is determined that the loop process from T110 to T115 has been completed for all events subsequent to the note-on event indicated by the pointer me (step T110; YES), the control section 11 proceeds to step T127.

[0074] On the other hand, in step T108, if it is determined that the damper pedal of pedal 28 was not turned ON before the note ON event indicated by pointer me (step T108; NO), the control unit 11 sets the information of pointer me to pointer next (step T116).

[0075] Next, the control unit 11 sets the pedal event flag (pedal_flag) to false (step T117).

[0076] Next, the control unit 11 judges whether or not the loop process from T118 to T126 has been completed for all events following the note-on event indicated by the pointer me (step T118). If the pointer "next" contains a number greater than the number of the last event in the performance information, the control unit 11 determines that the loop process from T118 to T126 has been completed for all events following the note-on event indicated by the pointer "me."

[0077] If it is determined that the loop processing from T118 to T126 has not been completed for all events after the note ON event indicated by the pointer me (step T118; NO), the control unit 11 determines whether or not there is a pedal ON event during the sounding of the note of the note ON event indicated by the pointer me (step T119).

[0078] When it is determined that a pedal ON event occurs during the sounding of the note of the note ON event indicated by the pointer me (step T119; YES), the control unit 11 determines whether or not the event indicated by the pointer next is a pedal ON event (step T120). When it is determined that the event indicated by the pointer "next" is a pedal-on event (step T120; YES), the control unit 11 sets "pedal_flag" to "true" (step T121), and proceeds to step T122. When it is determined that the event indicated by the pointer "next" is not a pedal-on event (step T120; NO), the control unit 11 proceeds to step T122.

[0079] In step T122, if the control unit 11 determines that pedal_flag is true and that the event indicated by the pointer next is a pedal OFF event and is outside the note of the note ON event indicated by the pointer me (step T122; YES), the control unit 11 calculates the difference between the timing of the note ON event indicated by the pointer next (next->lTime) and the timing of the note ON event indicated by the pointer me (me->lTime), adds the calculation result to the information of the note ON event indicated by the pointer me as the effective note length (me->pedal_gate) corresponding to the damper pedal operation of the pedal 28 of the note ON event indicated by the pointer me (step T123), and proceeds to step T127.

[0080] On the other hand, if pedal_flag is true and it is determined that the event indicated by pointer next is a pedal OFF event and is within the note of the note ON event indicated by pointer me (step T122; NO, step T124; YES), the control unit 11 adds the note length (me->lGate) of the note ON event indicated by pointer me to the information of the note ON event indicated by pointer me as the effective note length (me->pedal_gate) corresponding to the operation of pedal 28 of the note ON event indicated by pointer me (step T125), and proceeds to step T127.

[0081] If the event indicated by the pointer "next" is not a pedal-off event or "pedal_flag" is false (step T122; NO, step T124; NO), the control unit 11 advances the pointer "next" to the next number (step T126) and returns to step T118.

[0082] The control unit 11 repeatedly executes the loop process of steps T118 to T126 until it determines that the loop process of T118 to T126 has been completed for all events subsequent to the note-on event indicated by the pointer me. In step T118, when it is determined that the loop process from T118 to T126 has been completed for all events subsequent to the note-on event indicated by the pointer me (step T118; YES), the control section 11 proceeds to step T127.

[0083] In step T127, the control unit 11 advances the pointer me to the next number (step T127), and returns to step T103. The control unit 11 repeatedly executes the loop process of steps T103 to T127 until it determines that the loop process of steps T103 to T127 has been completed up to the last event of the performance information.

[0084] In step T103, if it is determined that the loop process of steps T103 to T127 has been completed up to the last event of the performance information (step T103; YES), the control unit 11 ends the process of obtaining the substantial note length according to the damper pedal operation. When the process of acquiring the substantial note lengths corresponding to the damper pedal operation for both the model performance information and the user performance information is completed, the control unit 11 proceeds to step S702 in FIG.

[0085] In step S702 of FIG. 5, the control unit 11 initializes the note pointer ne of the model performance (step S702). The pointer ne is a variable for storing information indicating the note-on event to be processed among the note-on events included in the model performance information (here, the number from the first note-on event in the model performance information). In step S702, the note pointer ne is set to 0, which is the number of the first note-on event.

[0086] Next, the control unit 11 judges whether or not the loop process from step S703 to step S710 has been completed up to the last event of the model performance information (step S703). If the note pointer ne contains a number greater than the number of the last note-ON event in the model performance information, the control unit 11 determines that the loop process from step S703 to step S710 has been completed up to the last event in the model performance information.

[0087] When it is determined that the loop process from step S703 to step S710 has not been completed up to the last event of the model performance information (step S703; NO), the control unit 11 initializes the note pointer me and the pointer pe of the user performance information (step S704). The note pointer me is a variable for storing information indicating a note ON event to be processed among the note ON events included in the user performance information (here, the number from the first note ON event). In step S704, the note pointer me is set to 0, which is the number of the first note ON event. The control unit 11 also sets null to the pointer pe, which is a variable for storing information indicating a note ON event of the user performance information corresponding to the note ON event indicated by the note pointer ne of the model performance information.

[0088] Next, the control unit 11 judges whether or not the loop process of steps S705 to S708 has been completed up to the last note-ON event of the user performance information (step S705). Here, if the note pointer me contains a number greater than the number of the last note-ON event in the user performance information, the control unit 11 determines that the loop process of steps S705 to S708 has been completed up to the last note-ON event in the user performance information.

[0089] If it is determined that the loop processing of steps S705 to S708 has not been completed up to the last note ON event of the user performance information (step S705; NO), the control unit 11 determines whether the note ON event indicated by the note pointer me is the closest note ON event of the same pitch as the note ON event indicated by the note pointer ne (step S706).

[0090] If it is determined that the note ON event indicated by the note pointer me is the closest note ON event at the same pitch as the note ON event indicated by the note pointer ne (step S706; YES), the control unit 11 sets the value of the note pointer me to the pointer pe, which is a variable that stores information indicating the note ON event of the user performance information corresponding to the note ON event indicated by the note pointer ne of the model performance information (step S707), and proceeds to step S708. If it is determined that the note ON event indicated by the note pointer me is not the closest note ON event of the same pitch as the note ON event indicated by the note pointer ne (step S706; NO), the control unit 11 proceeds to step S708.

[0091] In step S708, the control unit 11 advances the note pointer me to the next number (step S708), and returns to step S705. The control unit 11 repeatedly executes the loop process of steps S705 to S708 until the final note-ON event of the user performance information is reached and the loop process of steps S705 to S708 is completed.

[0092] If it is determined that the loop processing of steps S705 to S708 has been completed up to the last note ON event of the user performance information (step S705; YES), the control unit 11 calculates an index value indicating the difference in expression between the note ON event (performance operation) of the model performance information indicated by the note pointer ne and the note ON event (performance operation) of the user performance information indicated by the pointer pe, adds the calculation result to the information of the note ON event of the user performance information indicated by the pointer pe (step S709), and proceeds to step S710.

[0093] In step S709, the control unit 11 calculates the following index values ​​(1) to (4) using the following (Equation 1) to (Equation 4).

[0094] (1) An index value (iAnaDistance) indicating the difference in timing between the note-on of the model performance and the corresponding note-on of the user's performance pe->iAnaDistance=50 + (pe->lTime - ne->lTime) × 50 / quarter note length...(Formula 1) Here, pe->iAnaDistance indicates the iAnaDistance between the note-on event of the model performance information indicated by the note pointer ne and the note-on event of the user performance information indicated by the pointer pe. pe->lTime is the timing of the note-on event indicated by the pointer pe (time measurement in ticks). ne->lTime is the timing of the note-on event indicated by the note pointer ne (time measurement in ticks). pe->iAnaDistance is normalized by the length of a quarter note when the difference in the Note-ON timing between the user's performance and the model performance is, and if there is no difference between the Note-ON timing of the user's performance and the model performance (if they match), it will be 50. If the Note-ON timing of the user's performance is faster than the Note-ON timing of the model performance, pe->iAnaDistance will be a value greater than 50, and will be 100 if it is faster than the Note-ON timing of the model performance by the length of a quarter note. If the Note-ON timing of the user's performance is slower than the Note-ON timing of the model performance, pe->iAnaDistance will be a value less than 50, and will be 0 if it is slower than the Note-ON timing of the model performance by the length of a quarter note.

[0095] (2) An index value (iAnaDelta) indicating the difference in timing between the note-on of the model performance and the corresponding note-on of the user's performance pe->iAnaDelta = (pe->lTime - ne->lTime) …(Formula 2) Here, pe->iAnaDelta indicates the iAnaDelta between the Note ON event of the model performance information indicated by the note pointer ne and the Note ON event of the user performance information indicated by the pointer pe. iAnaDelta is an index used in the visualization process described later, and is the difference in note-on timing between the user's performance and the model performance.

[0096] (3) An index value (iAnaLength) indicating the difference in length between the note-on of the model performance and the corresponding note-on of the user's performance. pe->iAnaLength = 50 + 50 × ( ( pe->lGate - ne->lGate ) / pe->lGate ) …(Equation 3) Here, pe->iAnaLength indicates the iAnaLength between the Note ON event of the model performance information indicated by the note pointer ne and the Note ON event of the user performance information indicated by the pointer pe. pe->lGate is the note length (length in tick units) of the Note ON event indicated by the pointer pe. ne->lGate is the note length (length in tick units) of the Note ON event indicated by the note pointer ne. pe->iAnaLength is the difference between the note lengths of the note ON events in the user's performance and the model performance normalized by the note length of the user's performance (pe->lGate), and will be 50 if there is no difference between the note lengths of the note ON events in the user's performance and the model performance (if they match). If the note length of the user's performance is longer than that of the model performance, pe->iAnaLength will be a value greater than 50, and will be 100 if the note length of the user's performance is longer than the note ON timing of the model performance by the note length of the user's performance. If the note length of the user's performance is shorter than that of the model performance, pe-> pe->iAnaLength will be a value less than 50, and will be 0 if the note length of the user's performance is shorter than the note ON timing of the model performance by the note length of the user's performance.

[0097] (4) An index value (iALengthWithPedal) indicating the difference in effective note duration between the Note ON of the model performance and the corresponding Note ON of the user performance in response to damper pedal operation. pe->iALengthWithPedal = 100 × pe->pedal_gate / ne->pedal_gate…(Equation 4) pe->iALengthWithPedal indicates the iALengthWithPedal between the Note ON event of the model performance information indicated by the note pointer ne and the Note ON event of the user performance information indicated by the pointer pe. pe->iALengthWithPedal is a value that indicates, in percentage, the ratio of the effective note duration corresponding to the damper pedal operation in the user's performance to the effective note duration corresponding to the damper pedal operation in the model performance.

[0098] If there is no note-on event corresponding to the note-on event indicated by the note pointer ne in the user performance information (if a performance has been missed), that is, if the pointer pe is set to null, the process of step S709 is omitted.

[0099] Next, the control unit 11 advances the note pointer ne to the next number (step S710), and returns to step S703. The control unit 11 repeatedly executes the loop process of steps S703 to S710 until it determines that the loop process of steps S703 to S710 has been completed up to the last event of the model performance information.

[0100] When it is determined that the loop process from step S703 to step S710 has been completed up to the last event of the model performance information (step S703; YES), the control section 11 executes a process of smoothing the time change of the index value (step S711).

[0101] 8 is a flow chart showing the flow of the smoothing process of the time change of the index value. The smoothing process of the time change of the index value is executed by the control unit 11 and the performance assistance application 121 in cooperation with each other.

[0102] In the process of smoothing the time change of the index value, first, control unit 11 initializes lTick (step T201). Here, the smoothing process of the change in index value over time is performed in 32nd note units. This is because the index values ​​for almost all notes can be reflected by using 32nd note units. lTick is a variable that stores the timing (in ticks) that separates each 32nd note unit. In step T201, the control unit 11 sets the value of the length of a 32nd note (in ticks) that corresponds to the timing of the first 32nd note unit from the beginning as the initial value to lTick. Note that the unit of the smoothing process is not limited to 32nd note units.

[0103] Next, the control unit 11 initializes the note pointer ne of the model performance (step T202). The control unit 11 sets the note pointer ne to 0.

[0104] Next, the control unit 11 initializes the note pointer me of the user's performance (step T203). The control unit 11 sets the note pointer me to 0.

[0105] Next, the control unit 11 initializes the tempo information pointer mt (step T204). The pointer mt is a variable for storing information (here, a number from the beginning) indicating the tempo information to be processed among the tempo information included in the model performance information. In step T201, the pointer mt is set to 0, which is the number of the first tempo information.

[0106] Next, the control unit 11 initializes various variables (step T205). The control unit 11 sets, for example, w_avr_u, w_avr_g, w_avr_m, w_avr_mm, w_avr_p, and w_avr_t used in the following processes to an initial value of 0. In addition, the control unit 11 sets the bpm value of the first tempo information in the model performance information as the initial value to w_avr_bpm.

[0107] Next, the control unit 11 determines whether the smoothing process has been completed up to the end of the user's performance information and the model performance information (step T206).

[0108] If it is determined that the smoothing process has not been completed up to the end of the user performance information and the model performance information (step T206; NO), the control unit 11 determines whether the note-on timing (me->lTime) of the note-on event indicated by the note pointer me is less than lTick (step T207).

[0109] If it is determined that the note ON timing (me->lTime) of the note ON event indicated by the note pointer me is less than lTick (step T207; YES), the control unit 11 obtains the velocity value (strength value) of the note ON event indicated by the note pointer me and stores it in the variable value (step T208). Next, the control unit 11 calculates the weighted moving average value (w_avr_u) of the strength values ​​of the user's performance by the following (Equation 5) (step T209). w_avr_u = ( value × weight + w_avr_u × ( 1.0 - weight ) …(Equation 5) Here, "weight" is a weighting coefficient, and it is preferable that "weight" is set to, for example, 0.2 to 0.3. The control unit 11 stores the calculation result of w_avr_u in the variable w_avr_u, and stores the calculation result in the index value storage area of ​​the RAM in association with lTick. If w_avr_u corresponding to lTick is already stored in the index value storage area, it is overwritten with the current calculation result.

[0110] Next, the control unit 11 stores the iAnaLength (me->iAnaLength) of the note ON event indicated by the note pointer me in the variable value (step T210). Next, the control unit 11 calculates the weighted moving average value (w_avr_g) of me->iAnaLength by (Equation 6) (step T211). w_avr_g = (value × weight + w_avr_g × (1.0 - weight)) …(Equation 6) Here, "weight" is a weighting coefficient, and it is preferable that "weight" is set to, for example, 0.2 to 0.3. The control unit 11 stores the calculation result of w_avr_g in the variable w_avr_g, and stores the calculation result in the index value storage area of ​​the RAM in association with lTick. If w_avr_g corresponding to lTick is already stored in the index value storage area, it is overwritten with the current calculation result.

[0111] Next, the control unit 11 stores the iAnaDistance (me->iAnaDistance) of the note ON event indicated by the note pointer me in the variable value (step T212). Next, the control unit 11 calculates the weighted moving average value (w_avr_m) of me->iAnaDistance by (Equation 7) (step T213). w_avr_m = (value × weight + w_avr_m × (1.0 - weight)) …(Equation 7) Here, "weight" is a weighting coefficient, and it is preferable that "weight" is set to, for example, 0.2 to 0.3. The control unit 11 stores the calculation result of w_avr_m in the variable w_avr_m, and stores the calculation result in the index value storage area of ​​the RAM in association with lTick. If w_avr_m corresponding to lTick is already stored in the index value storage area, it is overwritten with the current calculation result.

[0112] Next, the control unit 11 obtains (calculates) the tempo (bpm value) of the note-ON event indicated by the note pointer me, that is, the tempo of the user's performance, and stores it in the variable "value" (step T214). For example, in the case of a 4 / 4 time signature, the control unit 11 calculates the tempo (bpm value) of the user's performance using the following (Equation 8). value = 240 / ( ( 240 / w_avr_bpm ) + ( me->iAnaDelta ) × ( 240 / w_avr_bpm ) / 1920) …(Equation 8) For example, if one beat is 480 (Tick), one measure in 4 / 4 time is 1920 (Tick). The tempo (bpm) is the number of beats in 60 seconds, so the length of one beat is 60 / bpm (seconds). In the case of 4 / 4 time, one measure has 4 beats, so the length of one measure is 4 × 60 / bpm = 240 / bpm (seconds). Therefore, the tempo (bpm) is 240 / (length of one measure). (Equation 8) is a formula for calculating the tempo of the user's performance from the length of one measure of the user's performance, which is obtained by adding or subtracting the difference between the length of the user's performance and the model performance ( me->iAnaDelta ), converted into seconds, to the length of one measure of the model performance ( 240 / w_avr_bpm ). For example, if me->iAnaDelta = -120, w_avr_bpm = 120bpm, and me->iAnaDelta converted to seconds is Δa, Δa = -120 × ( 240 / 120 ) / ( 480 × 4 ) = -120 × 2 / 1920 = -0.125 (seconds) value = 240 / ( ( 240 / w_avr_bpm ) + Δa )) = 240 / ( 2 - 0.125 ) = 128 bpm It becomes.

[0113] Next, the control unit 11 calculates the weighted moving average value (w_avr_mm) of the tempo of the user's performance by (Equation 9) (step T215). w_avr_mm = (value × weight + w_avr_mm × (1.0 - weight)) …(Equation 9) Here, "weight" is a weighting coefficient, and it is preferable that "weight" is set to, for example, 0.2 to 0.3. The control unit 11 stores the calculation result of w_avr_mm in the variable w_avr_mm, and stores the calculation result in the index value storage area of ​​the RAM in association with lTick. If w_avr_mm corresponding to lTick is already stored in the index value storage area, it is overwritten with the current calculation result.

[0114] Next, the control unit 11 stores iALengthWithPedal (me->iALengthWithPedal) of the note ON event indicated by the note pointer me in the variable value (step T216). Next, the control unit 11 calculates the weighted moving average value (w_avr_p) of me->iALengthWithPedal by (Equation 10) (step T217). w_avr_p = (value × weight + w_avr_p × (1.0 - weight)) …(Equation 10) Here, "weight" is a weighting coefficient, and it is preferable that "weight" is set to, for example, 0.2 to 0.3. The control unit 11 stores the calculation result of w_avr_p in the variable w_avr_p, and stores the calculation result in the index value storage area of ​​the RAM in association with lTick. If w_avr_p corresponding to lTick is already stored in the index value storage area, it is overwritten with the current calculation result.

[0115] Next, the control unit 11 advances the note pointer me to the next number (step T218), and returns to step T207. The control unit 11 repeatedly executes the processes of steps T207 to T218 while updating the note pointer me until it determines that the note-on timing (me->lTime) of the note-on event indicated by the note pointer me has exceeded lTick. In addition, for a Note ON event (such as a mistouch) that does not have a corresponding Note ON in the model performance information, iAnaDistance, iAnaDelta, iAnaLength, and iALengthWithPedal are not calculated, so the processes of T210 to T217 are omitted for Note ON events for which these index values ​​are not calculated.

[0116] In step T207, if it is determined that the note ON timing (me->lTime) of the note ON event indicated by the note pointer me has exceeded lTick (step T207; NO), the control unit 11 determines whether the note ON timing (ne->lTime) of the note ON event indicated by the note pointer ne is within lTick (step T219). If it is determined that the note-on timing (ne->lTime) of the note-on event indicated by the note pointer ne is within lTick (step T219; YES), the control unit 11 obtains the velocity value (strength value) of the note-on event indicated by the note pointer ne and stores it in the variable value (step T220). .

[0117] Next, the control unit 11 calculates the weighted moving average value (w_avr_t) of the strength of the model performance by the following (Equation 11) (step T221). w_avr_t = ( value × weight + w_avr_t × ( 1.0 - weight ) …(Equation 11) Here, "weight" is a weighting coefficient, and it is preferable that "weight" is set to, for example, 0.2 to 0.3. The control unit 11 stores the calculation result of w_avr_t in the variable w_avr_t, and stores the calculation result in the index value storage area of ​​the RAM in association with lTick. If w_avr_t corresponding to lTick is already stored in the index value storage area, it is overwritten with the current calculation result.

[0118] Next, the control unit 11 advances the note pointer ne to the next number (step T222), and returns to step T219. The control unit 11 repeatedly executes the loop process of steps T219 to T222 while updating the note pointer ne until it determines that the note-on timing (ne->lTime) of the note-on event indicated by the pointer ne has exceeded lTick.

[0119] In step T219, if it is determined that the note-on timing (ne->lTime) of the note-on event indicated by the note pointer ne has exceeded lTick (step T219; NO), the control unit 11 determines whether the timing of the tempo information indicated by the pointer mt (mt->lTime) is within lTick (step T223). If it is determined that the timing (mt->lTime) of the tempo information indicated by the pointer mt is within lTick (step T223; YES), the control unit 11 stores the tempo (bpm value) of the tempo information indicated by the pointer mt in the variable value (step T224). .

[0120] Next, the control unit 11 calculates the weighted moving average value (w_avr_bpm) of the tempo of the model performance by the following (Equation 12) (step T225). w_avr_bpm = ( value × weight + w_avr_ bpm × ( 1.0 - weight ) …(Equation 11) Here, "weight" is a weighting coefficient, and it is preferable that "weight" is set to, for example, 0.2 to 0.3. The control unit 11 stores the calculation result of w_avr_bpm in the variable w_avr_bpm, and stores the calculation result in the index value storage area of ​​the RAM in association with lTick. If w_avr_bpm corresponding to lTick is already stored in the index value storage area, it is overwritten with the current calculation result.

[0121] Next, the control unit 11 advances the pointer mt to the next number (step T226), and returns to step T223. The control unit 11 repeatedly executes the loop process of steps T223 to T226 while updating the pointer mt, until it determines that the timing of the tempo information indicated by the pointer mt (mt->lTime) has exceeded lTick.

[0122] In step T223, if it is determined that the timing of the tempo information indicated by the pointer mt (mt->lTime) has exceeded lTick (step T223; NO), the control unit 11 adds a value (Tick) of a 32nd note to lTick (step S227) and returns to step T206.

[0123] The control unit 11 repeatedly executes the processes of steps T206 to T227 until it determines in step T206 that the smoothing process has been completed up to the end of the user's performance information and the model performance information. In step T206, when it is determined that the smoothing has been completed for the entire user performance information and the model performance information (step T206; YES), the control unit 11 proceeds to step S712 in FIG.

[0124] In step S712 of FIG. 5, the control unit 11 plots each smoothed index value (weighted moving average value) stored in the index value storage area on a coordinate plane with the index value on the vertical axis and time (Tick) on the horizontal axis, thereby superimposing a graph showing the change in each index value over time and displaying it on the display unit 14 (step S712), and then returns to step S3 of FIG. 2. The control unit 11 causes the display unit 14 to display each graph in a different color so that the graphs can be distinguished from one another. In this embodiment, the control unit 11 plots each smoothed index value on the coordinate plane every lTick (in units of 32nd notes). In step S712, the control unit 11 causes the display unit 14 to display the time change of each index value, which indicates the difference in expression between the model performance and the user's performance, in order to visualize the change.

[0125] 9(a) and (b) are graphs showing the time change of the index values ​​of the difference in length between the model performance and the user's performance (w_avr_g), the difference in timing between the model performance and the user's performance (w_avr_m), the strength of the model performance (w_avr_t), and the strength of the user's performance (w_avr_u). In the graph shown in FIG. 9(a), the user is performing from time t1 to time t2, and since the current time is t2, for example, the user is not performing after time t2. Similarly, in the graph shown in FIG. 9(b), the user is performing from time t11 to time t13, and is not performing after t13.

[0126] In the graph shown in FIG. 9(a), the index value indicating the difference in length between the model performance and the user's performance from the time measurement t1 to the time measurement t2 exceeds 50. In other words, this graph shows that the user played legato (played longer than the model performance) from the time measurement t1 to the time measurement t2. In addition, in the graph shown in FIG. 9(a), the index value indicating the difference in timing between the model performance and the user's performance from the time measurement t1 to the time measurement t2 is below 50. In other words, this graph shows that the user's performance tended to lag behind the model performance from the time measurement t1 to the time measurement t2. In addition, by displaying the graphs of the strength (velocity value) of the model performance and the strength (velocity value) of the user's performance superimposed on each other, the difference can be visually identified. Note that each graph in FIG. 9(a) is displayed in a different color so that it can be identified as a different one.

[0127] In the graph shown in FIG. 9(b), the index value indicating the difference in length between the model performance and the user's performance from the time t11 to the time t12 is below 50. That is, from this graph, it can be seen that the user played staccato (played shorter than the model performance) from the time t11 to the time t12. Also, in the graph shown in FIG. 9(b), the index value indicating the difference in timing between the model performance and the user's performance from the time t11 to the time t13 is below 50. That is, from this graph, it can be seen that the user's performance tended to lag behind the model performance from the time t11 to the time t13. Also, by displaying the graphs of the strength (velocity value) of the model performance and the strength (velocity value) of the user's performance superimposed on each other, the difference can be visually identified. Note that each graph in FIG. 9(b) is displayed in a different color so that it can be identified as a different one.

[0128] FIG. 10 is a graph showing changes over time in the index values ​​of the model performance tempo (w_avr_bpm), the user's performance tempo (w_avr_mm), the model performance strength (w_avr_t), and the user's performance strength (w_avr_u). In the graph shown in FIG. 10, the tempo of the model performance and the user performance are roughly the same, but the fine tempo fluctuations of the model performance are not seen in the user performance. In other words, it can be seen that the user is playing at roughly the same tempo as the model performance, but is not able to express as many fine tempo fluctuations as the model performance. In addition, in the graph shown in FIG. 10, the strength of the model performance and the user performance are roughly the same, but the user performance shows finer fluctuations than the model performance. In other words, it can be seen that the user is playing at roughly the same strength as the model performance, but there is subtle variation in strength. Note that each graph in FIG. 10 is displayed in a different color so that they can be distinguished from each other.

[0129] Fig. 11 is a graph showing the time change of an index value (error value: w_avr_p) indicating the difference in substantial note duration in response to damper pedal operation between a model performance and a user performance when the damper pedal is operated. Note that the graph shown in Fig. 11 also shows a reference line for the model performance (time change when the model performance and the user performance match). In the graph shown in FIG. 11, the index value is significantly larger than the reference line at timing t31, which indicates that the user played a note without an instruction to operate the damper pedal by depressing the damper pedal of pedal 28. The index value is also larger than the reference line between timing t31 and timing t32, which indicates that the timing of releasing the damper pedal was inappropriate. Around timing t33, the index value roughly matches the reference line, which indicates that the note length is roughly the same as the model performance. Between timing t34 and timing t35, the index value is below the reference line, which indicates that the damper pedal was not depressed where it was required. Note that the graph and the reference line in FIG. 11 are displayed in different colors so that they can be distinguished from each other.

[0130] As shown in the graphs of FIG. 9 to FIG. 11, in the performance support process, the index values ​​(w_avr_g, w_avr_m, w_avr_p) indicating the difference in expression between the model performance and the user performance, and the smoothed time changes of the index values ​​(w_avr_t, w_avr_u, w_avr_bpm, w_avr_mm) regarding the expressions of the model performance and the user performance are displayed so as to be visualized as a graph. That is, the time changes of the discrete index values ​​calculated in units of performance operations are smoothed by the weighted moving average and displayed as a graph. Therefore, the user can visually confirm the difference in performance expression between the model performance and the user performance, which should be regarded as a continuous change as the song progresses, and can notice the difference in expression that would not be noticed by listening alone. In addition, the difference in performance expression of the entire song between the model performance and the user performance can be viewed from a bird's-eye view. In addition, the user can confirm the difference between his / her own performance result and the model performance from the viewpoint of performance expression, which can increase the motivation to practice.

[0131] Furthermore, in the case of a piece of music that requires damper pedal operation, the length of the sound produced may differ depending on whether or not the damper pedal is operated. Therefore, it is not sufficient to simply indicate the difference between the length (note length) of the performance operation by key between the model performance and the user's performance. In this embodiment, the index value indicating the difference in expression between the model performance and the user's performance in this embodiment includes an index value (w_avr_p) indicating the difference in substantial note length according to the damper pedal operation between the model performance and the user's performance. This allows the user to confirm the difference in performance expression, including the damper pedal operation, between the model performance and the user's performance.

[0132] Returning to FIG. 2, if it is determined in step S3 that an instruction to end the musical performance assistance application has been given (step S3: YES), the control unit 11 ends the user performance input process (step S8) and ends the musical performance assistance process.

[0133] As described above, the control unit 11 of the information processing device 1 derives a first smoothed processing value obtained by smoothing an index value relating to the performance information of the model performance and a second smoothed processing value obtained by smoothing an index value relating to the performance information of the user's performance, and displays the difference between the first smoothed processing value and the second smoothed processing value on the display unit 14. In addition, the control unit 11 of the information processing device 1 derives a difference index value indicating the difference in expression between the model performance and the user's performance based on the performance information of the model performance and the performance information of the user's performance, smoothes the derived difference index value, and displays the smoothed index value on the display unit 14 so as to visualize it. Therefore, the user can easily confirm the difference in performance expression between the model performance and the user's performance. For example, the user can visually confirm the difference in performance expression between the model performance and the user's performance, which should be perceived as a continuous change as the song progresses, and can notice differences in expression that would not be noticed by listening alone. In addition, the user can get an overview of the difference in performance expression of the entire song between the model performance and the user's performance. In addition, since the user can confirm the difference between his or her own performance result and the model performance from the perspective of performance expression, the motivation to practice can be increased.

[0134] For example, the control unit 11 derives a value indicating the difference in timing between the performance operation of the model performance and the corresponding performance operation of the user's performance as a difference index value. Therefore, the user can easily confirm the difference in timing of the performance operation, which is one element of performance expression, between the model performance and the user's performance.

[0135] In addition, for example, the control unit 11 derives a value indicating the difference between the length of the performance operation of the model performance and the length of the corresponding performance operation of the user's performance as a difference index value. Therefore, the user can easily confirm the difference between the model performance and the user's performance in the length of the performance operation, which is one element of performance expression.

[0136] Also, for example, the control unit 11 calculates a value indicating the difference between the substantial note length corresponding to the damper pedal operation of the sound produced by the performance operation of the model performance and the substantial note length corresponding to the damper pedal operation of the sound produced by the performance operation of the corresponding user performance as a difference index value. Therefore, the user can easily confirm the difference between the substantial note length corresponding to the damper pedal operation of the sound produced by the performance operation, which is one element of the performance expression, between the model performance and the user performance.

[0137] The contents of the description in the above embodiment and modifications are preferred examples of the program, information processing device, electronic musical instrument, and display method according to the present invention, and the present invention is not limited to these.

[0138] For example, in the above embodiment, the information processing device 1 and the electronic musical instrument 2 are separate entities, but the functions of the present invention implemented by the control unit 11 of the information processing device 1 may be provided in the electronic musical instrument 2. Also, the graph displayed in step S712 of FIG. 5 may be displayed on the display unit 25 of the electronic musical instrument 2.

[0139] Also, for example, in the above embodiment, the electronic musical instrument 2 is a keyboard instrument, but it may be another electronic musical instrument, such as an electric guitar, a MIDI violin, or the like.

[0140] In addition, in the above embodiment, an example was given of using the effective note length corresponding to the damper pedal operation as one element of performance expression, but it is also possible to use the effective note length corresponding to other pedal operations such as the sostenuto pedal or soft pedal.

[0141] Furthermore, it is not necessary to calculate or obtain index values ​​for all of the elements listed as elements of performance expression in the above embodiment and visualize their changes over time; it is also possible to calculate or obtain index values ​​for only some of the elements and visualize their changes over time. Furthermore, in this embodiment, for the "timing," "length," and "effective note length in response to damper pedal operation" of the performance operation, index values ​​(difference index values) indicating the difference in these values ​​between the model performance and the user's performance are calculated as weighted moving average values ​​corresponding to changes over time and graphed; for "strength" and "tempo," "strength" and "tempo" for the model performance and the user's performance, respectively, are used as index values ​​related to expression, and weighted moving average values ​​corresponding to changes over time of each index value (a first smoothed processing value obtained by smoothing the index value for the model performance, and a second smoothed processing value obtained by smoothing the index value for the user performance) are derived; and the changes over time of the weighted moving average values ​​of the index values ​​for the model performance and the user's performance are graphed and overlaid. As another embodiment, for the performance operations "strength" and "tempo", index values ​​showing the difference between these values ​​of the model performance and the user performance are calculated and the time change is graphed, and for "timing", "length", and "effective note length according to damper pedal operation", weighted moving average values ​​(first smoothed processing value, second smoothed processing value) according to the time change of each index value are derived as index values ​​related to expression for each of the model performance and the user performance, and the time change of the weighted average value of each index value of the model performance and the user performance may be graphed and superimposed. In addition to "strength" and "tempo", at least one index value of "performance timing", "pitch", "duration length of each key press", and "effective note length according to pedal operation" may be calculated as a weighted moving average value for each of the model performance and the user performance, and the respective graphs may be superimposed and displayed, so that the difference between the model performance and the user performance can be visually understood.

[0142] In the above embodiment, an example is disclosed in which a semiconductor memory such as a ROM or a hard disk is used as a computer-readable medium for the program according to the present invention, but the present invention is not limited to this example. As other computer-readable media, a hard disk, an SSD, or a portable recording medium such as a CD-ROM can be applied. In addition, a carrier wave can be applied as a medium for providing data of the program according to the present invention via a communication line.

[0143] In addition, the detailed configurations and operations of the devices that make up the performance expression visualization system may be modified as appropriate without departing from the spirit and scope of the invention. It should be noted that, in this disclosure, an "embodiment" means that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment. The appearances of the phrase "in an embodiment" in various places throughout this specification do not necessarily all refer to the same embodiment, but may. In this disclosure, for example, the phrase "at least one of A, B, and C" means "(A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C)" unless otherwise specified. Also, for example, if C is plural, the phrase "at least one of A, B, and C" means "(A), (B), (at least one or more C), (A and B), (A and at least one or more C), (B and at least one or more C), or (A, B, and at least one or more C)". When A is plural or B is plural, it is interpreted as being treated in the same manner as above. The term "or" is used to refer to non-exclusive things unless otherwise stated, for example, the phrase "A or B" means "any of (A but not B), (B but not A), and (A and B)."

[0144] Although the embodiment of the present invention has been described above, the technical scope of the present invention is not limited to the above embodiment, but is defined based on the claims. Furthermore, the technical scope of the present invention also includes equivalent ranges that are made by adding modifications that are not related to the essence of the present invention from the claims. [Explanation of symbols]

[0145] 100 Performance Expression Visualization System 1. Information processing device 11 Control section 12 Storage section 121 Performance Support App 13 Input section 14 Display section 15 Communications Department 16 Output section 17 Bus 2. Electronic Instruments 21 Control section 22 Memory section 23 keys 24 Input section 25 Display section 26 Communications Department 27 Output section 28 Pedals 29 Bus

Claims

1. Computer, deriving a first smoothed value obtained by smoothing an index value relating to the performance information of the model performance and a second smoothed value obtained by smoothing an index value relating to the performance information of the user's performance; displaying a difference between the first smoothed value and the second smoothed value on a display unit; A program to function as a control unit.

2. The smoothing process is a weighted moving average according to time change, The control unit is displaying the difference on the display unit by overlaying the graph of the first smoothed value and the graph of the second smoothed value; The program according to claim 1.

3. The index value relating to the performance information includes at least one of a velocity value, a tempo, a performance timing, a tone pitch, a duration of each key depression, and a substantial tone duration according to a pedal operation. The program according to claim 1.

4. The control unit is displaying the graph of the first smoothed value and the graph of the second smoothed value on the display unit so that they can be distinguished from each other; The program according to claim 1.

5. Computer, deriving a difference index value indicating a difference in expression between the model performance and the user's performance based on performance information of the model performance and performance information of the user's performance; smoothing the derived difference index value over time; Displaying the smoothed difference index value on a display unit. A program to function as a control unit.

6. The control unit is deriving, as the difference index value, a value indicating a difference in timing between a performance operation of the model performance and a corresponding performance operation of the user's performance; The program according to claim 5.

7. The control unit is deriving, as the difference index value, a value indicating a difference between a length of a performance operation of the model performance and a length of a corresponding performance operation of the user's performance; The program according to claim 5.

8. The control unit is deriving, as the difference index value, a value indicating a difference between a substantial note duration corresponding to a pedal operation of the sound produced by the performance operation of the model performance and a substantial note duration corresponding to a pedal operation of the sound produced by the corresponding performance operation of the user performance; The program according to claim 5.

9. The control unit further includes: deriving a first smoothed value by smoothing an index value relating to the performance information of the model performance and a second smoothed value by smoothing an index value relating to the performance information of the user's performance; displaying a difference between the first smoothed value and the second smoothed value on a display unit together with the difference index value; The program according to claim 5.

10. the smoothed index values ​​as the first smoothed value and the second smoothed value include at least one of a velocity value, a tempo, a performance timing, a tone pitch, a duration of each key depression, and a substantial tone duration according to a pedal operation; The program according to claim 9.

11. deriving a first smoothed value obtained by smoothing an index value relating to the performance information of the model performance and a second smoothed value obtained by smoothing an index value relating to the performance information of the user's performance; displaying a difference between the first smoothed value and the second smoothed value on a display unit; An information processing device having a control unit.

12. deriving a difference index value indicating a difference in expression between the model performance and the user's performance based on performance information of the model performance and performance information of the user's performance; smoothing the derived difference index value over time; Displaying the smoothed difference index value on a display unit. An information processing device having a control unit.

13. The display unit is further provided. The information processing device according to claim 11 or 12.

14. An information processing device according to claim 11 or 12; a performance operator for performing the user performance; An electronic musical instrument comprising:

15. The display unit is further provided.

15. An electronic musical instrument according to claim 14.

16. The computer deriving a first smoothed value obtained by smoothing an index value relating to the performance information of the model performance and a second smoothed value obtained by smoothing an index value relating to the performance information of the user's performance; displaying a difference between the first smoothed value and the second smoothed value on a display unit; Display method.

17. The computer deriving a difference index value indicating a difference in expression between the model performance and the user's performance based on performance information of the model performance and performance information of the user's performance; smoothing the derived difference index value over time; Displaying the smoothed difference index value on a display unit. Display method.

Images