Program, performance evaluation device, and performance evaluation method

The performance evaluation device addresses the lack of tempo consideration in existing systems by allowing for the assessment and improvement of musical performances at specified tempos through tempo information acquisition and comparison with reference standards.

JP7585761B2Active Publication Date: 2024-11-19JVC KENWOOD CORP
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
JP2020208260
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2020-12-16
Publication Date
2024-11-19
Estimated Expiration
2040-12-16

AI Technical Summary

Technical Problem

Existing systems for evaluating musical instrument performance do not account for tempo, limiting the ability to assess performances at desired tempos.

Method used

A performance evaluation device that acquires tempo information and compares user performance with reference information to generate evaluation information, allowing for the assessment of musical performances at specified tempos.

Benefits of technology

Enables the evaluation of musical performances at desired tempos, providing a comprehensive assessment of performance quality and guiding users in improving their tempo control.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007585761000001
    Figure 0007585761000001
  • Figure 0007585761000002
    Figure 0007585761000002
  • Figure 0007585761000003
    Figure 0007585761000003
Patent Text Reader

Abstract

To evaluate a musical performance made on a musical instrument at a desired tempo.SOLUTION: A musical performance evaluation device 2000 acquires tempo information 100. The tempo information 100 is information for specifying a specific tempo as a tempo of a musical performance that a user 60 desires. The musical performance evaluation device 2000 acquires reference information 40 corresponding to the specific tempo for an object piece of music. The reference information 40 corresponding to the specific tempo represents an evaluation standard for a musical performance in which the piece of music to be evaluated is played at the specific tempo. The musical performance evaluation device 2000 acquires user musical performance information 50 representing a musical performance of the object piece of music by the user 60. The musical performance evaluation device 2000 compares the user musical performance information 50 with the reference information 40 to generate evaluation information 30.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a technique for evaluating musical instrument performance. [Background technology]

[0002] Systems for evaluating musical instrument performance have been developed. Patent Documents 1 and 2 disclose systems that photograph a performance of a keyboard instrument from above and compare the photographed performance with a model performance for evaluation. The system of Patent Document 1 evaluates the position, inclination, or width of the hands. On the other hand, the system of Patent Document 2 evaluates whether the fingering used in performance is correct. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2006-091633 A [Patent Document 2] JP 2002-182553 A Summary of the Invention [Problem to be solved by the invention]

[0004] Neither Patent Document 1 nor Patent Document 2 refers to the tempo of the performance. The present invention has been made to solve such problems, and its purpose is to provide a technique for evaluating a performance performed at a desired tempo. [Means for solving the problem]

[0005] The program disclosed herein causes a computer to execute a tempo information acquisition process that acquires tempo information capable of identifying a designated tempo, which is the performance tempo desired by the user, and an evaluation process that generates evaluation information including an evaluation of the performance by the user by comparing, for a piece of music to be evaluated, standard information that represents a performance evaluation standard when the piece is played at the designated tempo with user performance information that represents the performance of the piece of music by the user.

[0006] The performance evaluation device disclosed herein has a tempo information acquisition unit that acquires tempo information capable of identifying a designated tempo, which is the performance tempo desired by the user, and an evaluation unit that generates evaluation information including an evaluation of the performance by the user by comparing, for the song to be evaluated, standard information that represents a performance evaluation standard when played at the designated tempo with user performance information that represents the user's performance of the song.

[0007] The performance evaluation method disclosed herein is executed by a computer and includes a tempo information acquisition step of acquiring tempo information capable of identifying a designated tempo, which is a performance tempo desired by a user, and an evaluation step of generating evaluation information including an evaluation of the performance by the user by comparing, for a piece of music to be evaluated, standard information representing an evaluation standard for a performance when the piece of music is played at the designated tempo with user performance information representing the performance of the piece of music by the user. Effect of the Invention

[0008] According to the present invention, a technique is provided for evaluating a performance performed at a desired tempo. [Brief description of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating an example of an overview of a performance evaluation device according to a first embodiment. [Diagram 2] 1 is a block diagram illustrating an example of the functional configuration of a performance evaluation device according to a first embodiment. [Diagram 3]1 is a block diagram illustrating an example of the hardware configuration of a computer that realizes a performance evaluation device. [Figure 4] FIG. 2 is a diagram illustrating an example of the position and orientation of a camera. [Diagram 5] FIG. 2 is a diagram illustrating an example of joint points and joint lines of a hand. [Figure 6] FIG. 13 is a diagram illustrating a method for calculating the height of a finger joint. [Figure 7] 13 is a diagram illustrating a method for calculating the size of an angle between joint lines; FIG. [Figure 8] FIG. 13 is a diagram illustrating a method for calculating the height of the wrist. [Figure 9] 11 is a diagram illustrating a method for calculating the height of the hand upper surface. FIG. [Figure 10] 11 is a diagram illustrating a method for calculating a wrist angle. FIG. [Figure 11] FIG. 13 is a diagram illustrating a method for calculating the depth of a keystroke. [Figure 12] FIG. 1 is a diagram illustrating an example of a usage mode of a performance evaluation device. [Figure 13] 4 is a flowchart illustrating a process executed by the performance evaluation device of the first embodiment. [Figure 14] FIG. 13 is a diagram illustrating an example of a tempo designation screen. [Figure 15] FIG. 11 is a diagram illustrating an example of reference information in a table format. [Figure 16] FIG. 13 is a diagram illustrating an example of a designation screen for designating a target song. [Figure 17] 11 is a diagram illustrating an example of user performance information in a table format. FIG. [Figure 18] 13 is a flowchart illustrating a process of calculating an evaluation score for each item information item. [Figure 19] FIG. 11 is a diagram illustrating an example of evaluation information in a table format. [Figure 20] 13 is a diagram illustrating an example of evaluation information showing a statistical score for each performance time point. FIG. [Figure 21] FIG. 13 is a diagram illustrating an example of evaluation information indicating a statistical score for each evaluation item. [Figure 22]13 is a diagram illustrating an example of evaluation information indicating statistical scores for each partial range and for each evaluation item. FIG. [Diagram 23] FIG. 13 is a diagram illustrating an example of statistical values ​​of evaluation scores tallied for each group. [Figure 24] FIG. 11 is a first diagram illustrating an evaluation result screen. [Diagram 25] FIG. 2 is a second diagram illustrating an example of the evaluation result screen. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same or corresponding elements are given the same reference numerals, and duplicated explanations will be omitted as necessary for clarity of explanation. Furthermore, unless otherwise specified, predetermined values ​​such as predetermined values ​​and threshold values ​​are stored in advance in a storage device accessible from a device that uses the values.

[0011] [Embodiment 1] <Summary> Fig. 1 is a diagram illustrating an example of an outline of a performance evaluation device 2000 of embodiment 1. Fig. 1 is a diagram conceptually illustrating an example of the operation of the performance evaluation device 2000 to facilitate understanding of the performance evaluation device 2000, and the operation of the performance evaluation device 2000 is not limited to that shown in Fig. 1.

[0012] The performance evaluation device 2000 generates evaluation information 30 representing an evaluation of an instrument performance performed by a user 60. Here, the instrument played by the user 60 is called a target instrument 70. The target instrument 70 is, for example, a keyboard instrument such as a piano, an organ, or a synthesizer. However, the target instrument 70 is not limited to a keyboard instrument.

[0013] The performance evaluation device 2000 generates evaluation information 30 by comparing the actual performance given by the user 60 with a predetermined standard. The comparison is performed for one or more evaluation items. Hereinafter, the user 60 can specify the tempo of the performance when evaluating the performance of the piece of music to be evaluated. Hereinafter, the piece of music to be evaluated is referred to as the target piece of music. Also, the specified tempo of the performance is referred to as the specified tempo.

[0014] In order to evaluate the performance in the above-mentioned manner, the performance evaluation device 2000 uses tempo information 100, reference information 40, and user performance information 50. The tempo information 100 is information that can specify the specified tempo. The reference information 40 indicates the evaluation standard of the performance when the target musical piece is played at the specified tempo indicated in the tempo information 100. In other words, the reference information 40 indicates what kind of performance should be performed at what timing when the target musical piece is played at the specified tempo indicated in the tempo information 100. For example, the reference information 40 indicates the performance time and the ideal item value of the performance at the performance time for each of a plurality of evaluation items. The user performance information 50 is information that represents the actual performance of the target musical piece by the user. In other words, the user performance information 50 indicates what kind of performance the user 60 performed at what timing for the target musical piece. For example, the user performance information 50 indicates the performance time and the item value (actual value) of the performance of the user 60 at the performance time for each of a plurality of evaluation items.

[0015] The performance evaluation device 2000 generates evaluation information 30 by comparing the reference information 40 with the user performance information 50. For example, as described below, the performance evaluation device 2000 compares item values ​​indicated by the reference information 40 with item values ​​indicated by the user performance information 50 for each of a plurality of time points to determine the degree of match between the reference information 40 and the user performance information 50, and generates evaluation information 30 indicating the degree of match.

[0016] 1 illustrates an example in which the reference information 40 and the user performance information 50 are obtained from outside the performance evaluation device 2000. However, as will be described later, the reference information 40 and the user performance information 50 may be generated inside the performance evaluation device 2000. Methods of obtaining and generating the reference information 40 and the user performance information 50 will be described later in detail.

[0017] <Example of action and effect> According to the performance evaluation device 2000 of this embodiment, reference information 40 is acquired that indicates an evaluation standard when the target piece of music is performed at a specified tempo. Then, the reference information 40 is compared with user performance information 50 that indicates the result of the actual performance by the user 60, thereby evaluating the performance of the target piece of music by the user 60. Thus, according to the performance evaluation device 2000, it is possible to evaluate a performance of the target piece of music performed at a desired tempo. Thus, the user 60 can play the target piece of music at a desired tempo and receive an evaluation of the performance.

[0018] For example, the user 60 can practice the target piece of music while gradually increasing the tempo of the performance. First, the user 60 practices playing the target piece of music at a tempo slower than the standard tempo. Then, when the user 60 is able to obtain a sufficiently high evaluation (e.g., an evaluation higher than the evaluation set as a target) in the evaluation by the performance evaluation device 2000 specifying the specified tempo, the user 60 speeds up the tempo a little and practices the target piece of music further. By repeating such practice and evaluation while increasing the tempo, the user 60 can perform the target piece of music at the standard tempo or a tempo faster than the standard tempo and receive a high evaluation.

[0019] The performance evaluation device 2000 of this embodiment will be described in more detail below.

[0020] <Example of functional configuration> 2 is a block diagram illustrating an example of the functional configuration of a performance evaluation device 2000 of embodiment 1. In this example, the performance evaluation device 2000 has a tempo information acquisition unit 2020 and an evaluation unit 2040. The tempo information acquisition unit 2020 acquires tempo information 100. The evaluation unit 2040 generates evaluation information 30 by comparing reference information 40 corresponding to a designated tempo indicated in the tempo information 100 with user performance information 50.

[0021] <Example of hardware configuration> Each functional component of the performance evaluation device 2000 may be realized by hardware that realizes each functional component (e.g., hardwired electronic circuitry, etc.), or by a combination of hardware and software (e.g., a combination of an electronic circuitry and a program that controls it, etc.). Below, a further explanation will be given of the case where each functional component of the performance evaluation device 2000 is realized by a combination of hardware and software.

[0022] FIG. 3 is a block diagram illustrating an example of a hardware configuration of a computer 1000 that realizes the performance evaluation device 2000. The computer 1000 is any computer. For example, the computer 1000 is a stationary computer such as a PC (Personal Computer) or a server machine. Alternatively, the computer 1000 is a portable computer such as a smartphone or a tablet terminal. Alternatively, the computer 1000 may be a camera 10. Alternatively, for example, if the instrument played by the user 60 is an electronic instrument, the computer 1000 may be a computer built into the instrument.

[0023] The computer 1000 may be a dedicated computer designed to realize the performance evaluation device 2000, or may be a general-purpose computer. For example, the functions of the performance evaluation device 2000 are realized by the computer 1000 by installing a specific application on the computer 1000. The application is constituted by a program for realizing the functional components of the performance evaluation device 2000.

[0024] The computer 1000 has a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input / output interface 1100, and a network interface 1120. The bus 1020 is a data transmission path for the processor 1040, the memory 1060, the storage device 1080, the input / output interface 1100, and the network interface 1120 to transmit and receive data to and from each other. However, the method of connecting the processor 1040 and the like to each other is not limited to a bus connection.

[0025] The processor 1040 is a processor such as a central processing unit (CPU), a graphics processing unit (GPU), or a field-programmable gate array (FPGA). The memory 1060 is a main storage device realized using a random access memory (RAM) or the like. The storage device 1080 is an auxiliary storage device realized using a hard disk, a solid state drive (SSD), a memory card, or a read only memory (ROM) or the like.

[0026] The input / output interface 1100 is an interface for connecting the computer 1000 to an input / output device. The network interface 1120 is an interface for connecting the computer 1000 to a network. This network may be a LAN (Local Area Network) or a WAN (Wide Area Network).

[0027] The storage device 1080 stores a program (a program that realizes the above-mentioned application) that realizes each functional component of the performance evaluation device 2000. The processor 1040 reads this program into the memory 1060 and executes it to realize each functional component of the performance evaluation device 2000.

[0028] The performance evaluation device 2000 may be realized by one computer 1000, or may be realized by multiple computers 1000. In the latter case, the configurations of the computers 1000 do not need to be the same, and may be different from each other.

[0029] <Evaluation items> The performance by the user 60 is evaluated based on a number of evaluation items. The evaluation items may be any items that reflect the quality of the performance. Such evaluation items include, for example, items related to sound characteristics (such as sound intensity and duration). Other evaluation items include, for example, the shape of the hands and fingers when playing, fingering, and the shape of the target instrument when played. Here, fingering refers to the correctness of the correspondence between the pressed keys and the fingers. Also, the shape of the target instrument when played refers to, for example, the angle and depth of the keys when pressed with the fingers in the case of a keyboard instrument.

[0030] When items related to sound characteristics are included in the evaluation items, for example, a recorder is installed to record the performance of user 60, and the performance is evaluated using the audio data generated by the recorder. However, the audio data may be generated together with the video data by a camera, which will be described later. In other words, in this case, the camera also plays the role of a recorder. For example, when a camera generates a video file including video data and audio data, the audio data included in the video data can be used to evaluate the sound characteristics (evaluate whether the sound is good or bad).

[0031] Alternatively, for example, the target instrument 70 may be provided with a function for generating performance data for a performance performed using the target instrument 70. The performance data may be, for example, MIDI (registered trademark) (Musical Instrument Digital Interface) data. When performance data is generated by the target instrument 70 in this way, the performance data can be used to evaluate the characteristics of the sound.

[0032] When the evaluation items include items that can be evaluated using visual information, for example, a camera is installed to capture the performance of the user 60, and the performance of the user 60 is evaluated using video data generated by the camera. Such evaluation items include, for example, the shape of the hands and fingers when playing, the fingering, and the shape of the target instrument when playing. When the performance of the user 60 is evaluated for each of the left and right hands, for example, it is preferable to provide both a camera installed at a position where the left hand of the user 60 is in front and a camera installed at a position where the right hand of the user 60 is in front.

[0033] Fig. 4 is a diagram illustrating the position and orientation of the camera 10 capturing an image of a performance by the user 60. Fig. 4(a) is a plan view of the target instrument 70, and Fig. 4(b) is a front view of the target instrument 70. In Fig. 4, the longitudinal direction of the keyboard of the target instrument 70 is the x-axis direction, the lateral direction of the keyboard of the target instrument 70 is the y-axis direction, and the vertical direction is the z-axis direction.

[0034] Camera 10-1 is set up so as to capture at least the side of the fingers of the left hand of user 60 from the left side (from a direction in which the left hand is closer to the viewer than the right hand). In Fig. 4, camera 10-1 is set up so as to look down on target musical instrument 70 in the +x direction.

[0035] On the other hand, camera 10-2 is installed so as to capture at least the side of the fingers of the right hand of user 60 from the right side (from a direction in which the right hand is closer to the viewer than the left hand). In Fig. 4, camera 10-2 is installed so as to look down on target musical instrument 70 in the -x direction.

[0036] The shooting direction of the camera 10 is not limited to the example shown in Fig. 4. For example, the horizontal components of the shooting directions of the cameras 10-1 and 10-2 do not need to be completely parallel to the x-axis, and may include a component in the y-axis direction.

[0037] The specific installation position and installation orientation of the camera 10 may be determined arbitrarily by the user or may be determined in advance. In the former case, the camera 10 is installed on or around the musical instrument in a position and orientation that allows the performance of the user 60 to be photographed from the side.

[0038] On the other hand, when the installation position and installation orientation of the camera 10 are predetermined, for example, a tool is used that can be installed at a predetermined position on the target instrument and that can install the camera 10 in a predetermined orientation. This tool may be installed on the target instrument when the target instrument is provided (for example, sold or leased), or may be installed after the target instrument is provided (for example, provided as a separately sold attachment and installed by the user).

[0039] Alternatively, for example, the camera 10 may be installed at a predetermined position and attitude by adjusting the installation position and attitude of the camera 10 while viewing video data generated by the camera 10. For example, a predetermined reference mark is provided at a predetermined position on the target musical instrument. Then, the reference mark provided on the target musical instrument is photographed by the camera 10, and while viewing the video data obtained from the camera 10, the position and attitude of the camera 10 are adjusted so that the reference mark is photographed by the camera 10 at a predetermined position, size, and shape.

[0040] In the above example, the camera 10-1 and the camera 10-2 capture images of different hands, but this is not necessarily required. For example, the camera 10-1 captures images of some of the fingers of the left hand and some of the fingers of the right hand, and the camera 10-2 captures images of the remaining fingers of the left hand and the right hand. In other words, the video data 20 obtained from the camera 10-1 is used to evaluate some of the fingers of the left hand and some of the fingers of the right hand, and the video data 20 obtained from the camera 10-2 is used to evaluate the remaining fingers of the left hand and the right hand. As a more specific example, the video data 20 obtained from the camera 10-1 is used to evaluate the middle finger, ring finger, and little finger of the left hand and the thumb and index finger of the right hand, and the video data 20 obtained from the camera 10-2 is used to evaluate the thumb and index finger of the left hand and the middle finger, ring finger, and little finger of the right hand. In this way, even if one hand is an obstacle to imaging the other hand and therefore all the fingers on one hand cannot be imaged by one camera 10 alone, by using the video data 20 obtained from the two cameras 10, it is possible to evaluate all the fingers on both the left and right hands.

[0041] Furthermore, the number of cameras 10 used to evaluate a performance is not limited to two. For example, when evaluating the performance of the user 60 for only one hand, one camera 10 capable of photographing the hand to be evaluated may be provided. Alternatively, for example, multiple cameras 10 may be provided for each of the right and left hands so that the same hand is photographed at different angles of depression.

[0042] The camera 10 is any camera capable of capturing a performance by the user 60 and generating video data. The camera 10 may be a dedicated camera for capturing a performance of a musical instrument by the user, or may be a general-purpose camera (for example, a camera provided in a smartphone).

[0043] For example, the camera 10 is connected to a computer 1000 that realizes the performance evaluation device 2000 via an input / output interface 1100 or a network interface 1120. However, the computer 1000 is only required to be able to acquire video data generated by the camera 10, and does not have to be communicatively connected to the camera 10. A method for the performance evaluation device 2000 to acquire video data will be described later. The performance evaluation device 2000 may also be realized by the camera 10. In other words, the performance evaluation device 2000 is realized by a computer (for example, a semiconductor chip such as a System on Chip (SoC)) provided in the camera 10.

[0044] As described above, evaluation items for which video data may be used for the evaluation include evaluation items related to the shape of the hand and fingers. The item values ​​of these evaluation items are calculated using, for example, the joint points and joint lines of the hand. A joint point is a point that indicates the position of a joint. A joint line is a straight line that connects adjacent joint points. In addition, a straight line that connects a fingertip and a joint point of a joint closest to the fingertip (i.e., the first joint) is also treated as a joint line. FIG. 5 is a diagram illustrating examples of the joint points and joint lines of a hand. In FIG. 5, the joint points are represented by circles.

[0045] Below, some specific examples of evaluation items whose item values ​​are calculated based on the joint points and joint lines of the hand will be given.

[0046] <<Knuckle height>> "Finger joint height" is an evaluation item that indicates the position of the finger joint in the height direction. The joint points used by the performance evaluation device 2000 for evaluation may be all joint points or only some of them.

[0047] For example, the height of a finger joint is determined by the positional relationship between the joint point of the finger and a reference plane. For example, the reference plane is determined by the top surface of a keyboard. For example, the top surface of a keyboard in a normal state where the key is not being pressed is treated as the reference plane. Alternatively, for example, the top surface of a keyboard in a state where the key is pressed all the way down may be treated as the reference plane. Various methods can be used as a specific method for calculating the height of a finger joint.

[0048] The knuckle height is preferably calculated for fingers that are in a typing state, however, the knuckle height may also be calculated for fingers that are not in a typing state.

[0049] Fig. 6 is a diagram illustrating a method for calculating the height of a finger joint. For ease of explanation, Fig. 6 shows only the image area of ​​the video frame 22 around the finger 110 and the key 120 being pressed by the finger 110.

[0050] In FIG. 6, a reference line 130 is used to indicate the position of the reference plane. The reference line 130 is a straight line overlapping with the long side of the key adjacent to the key 120 on which the finger 110 is placed, which is farther from the camera 10. It is preferable to use the long side of the top surface of the key 120 in an undepressed state as the reference line. The method of indicating the position of the reference plane is not limited to the method using the reference line 130. For example, the long side of the key adjacent to the key 120, which is closer to the camera 10, may be used as the reference line indicating the position of the reference plane.

[0051] 6(a) and (b) show different methods for calculating the height of a finger joint. In FIG. 6(a), the height of joint point X is represented by the length of a line segment XA. Point A is the intersection point between a straight line extending vertically downward from joint point X on the video frame 22 and the reference line 130. On the other hand, in FIG. 6(b), the height of joint point X is represented by the length of a line segment XB. Point B is the foot of a perpendicular line drawn from joint point X to the reference line 130.

[0052] In the example of FIG. 6, the camera 10 is installed so as to capture an image of the side of the hand from diagonally above. In this case, the vertical direction of the video frame 22 and the direction of the perpendicular line drawn to the reference line 130 in FIG. 6(b) are different from the vertical direction in the real world. However, as shown in FIG. 6, the item value of the evaluation item "fingertip height" need only represent the distance from the joint point to the reference plane in a direction that includes a vertical component (i.e., a direction that is not horizontal), not the height in the vertical direction in the real world. The same applies to the "wrist height" described later.

[0053] However, the performance evaluation device 2000 may treat the height of a finger joint as the height in the vertical direction in the real world. In this case, for example, the performance evaluation device 2000 calculates the position and reference plane of the finger joint in a three-dimensional coordinate system of the real world (hereinafter, world coordinate system) from the position and reference plane of the finger joint in a two-dimensional coordinate system (hereinafter, image coordinate system) on the video frame 22, thereby calculating the height of the finger joint in the world coordinate system.

[0054] Conversion from the coordinates of the two-dimensional coordinate system to the coordinates of the three-dimensional coordinate system can be realized, for example, by using stereo vision or a depth camera. In the former case, two cameras 10 are provided, one for shooting the performance of the user 60 from the left side and the other for shooting the performance from the right side. Then, to calculate the height of the finger joints of the right hand of the user 60, the height of the finger joints on the world coordinate system is calculated using the video frames 22 obtained from each of the two cameras 10 shooting the user 60 from the right side. When a depth camera is used, the distance to the photographed object is known for each pixel on the video frame 22. Therefore, the three-dimensional coordinates of the finger joints in the world coordinate system can be specified from the two-dimensional coordinates of the finger joints in one video frame 22.

[0055] In order to determine the world coordinate system, it is preferable to provide a predetermined mark (a mark of a certain size and shape) that serves as a reference for the world coordinate system at a certain position on the target musical instrument. By photographing a mark of known size and shape with a camera, existing technology can be used to determine the relationship between the two-dimensional coordinate system on the image obtained from the camera and the world coordinate system.

[0056] <<The size of the angle between the joint lines>> "Amount of angle between joint lines" is an evaluation item that represents the size of an angle formed by two joint lines that have a common end point. FIG. 7 is a diagram illustrating a method for calculating the size of an angle between joint lines. For ease of explanation, FIG. 7 illustrates only an image area around a finger 110 in a video frame 22. Note that it is preferable that the size of the angle between joint lines is calculated for a finger that is typing. However, the size of the angle between joint lines may also be calculated for a finger that is not typing.

[0057] In FIG. 7, joint point C and joint point D are adjacent to each other, and the straight line connecting these joint points is joint line CD. Similarly, joint point D and joint point E are adjacent to each other, and the straight line connecting these joint points is joint line DE. The magnitude of the angle formed by these joint lines CD and DE is α. Therefore, for example, this value α can be used as an item value of an evaluation item of the magnitude of the angle formed by joint line CD and joint line DE. In addition, a joint line EF connecting joint line E and fingertip point F may be further used to provide an evaluation item of the magnitude of the angle formed by joint line DE and joint line EF.

[0058] Alternatively, for example, the value of the item "angle between two joint lines" may be the size of the angle between these two joint lines in the world coordinate system. In this case, the performance evaluation device 2000 identifies the two joint lines in the world coordinate system by converting the coordinates of each finger joint on the video frame 22 into coordinates in the world coordinate system. In this way, the size of the angle between the two joint lines in the world coordinate system can be identified.

[0059] <<Wrist height>> "Wrist height" is an evaluation item that indicates the position of the wrist in the height direction. For example, the wrist height is determined by the positional relationship between the wrist joint point and a reference plane. This reference plane can be the same as the reference plane that determines the height of the finger joints. Note that the wrist height is preferably calculated for a hand in which at least one finger is typing. However, the wrist height may also be calculated for a hand in which none of the fingers are typing.

[0060] 8 is a diagram illustrating a method for calculating the wrist height. For ease of explanation, in FIG. 8, only the image area of ​​the video frame 22 surrounding the hand 140 and the key that the hand 140 is typing is shown.

[0061] In FIG. 8, the wrist joint point of the hand 140 is Y. In FIG. 8, a reference line 150 is used to indicate the position of the reference plane. The reference line 150 is a straight line that overlaps with the long side of the key adjacent to the pressed key 160, which is farther from the camera 10. As in the case of calculating the height of the finger joints, it is also possible to use the long side of the top surface of the key when it is not being pressed as the reference line. In addition, the key used for the reference line is not limited to the key pressed by the finger closest to the camera 10 or the adjacent key.

[0062] 8(a) and (b) show different methods for calculating the wrist height. In FIG. 8(a), the wrist height is determined by the distance of a line segment YG. Point G is the intersection point of a straight line drawn vertically downward from the wrist joint point Y on the video frame 22 and the reference line 150. On the other hand, in FIG. 8(b), the wrist height is determined by the distance of a line segment YH. Point H is the foot of a perpendicular line drawn from the wrist joint point Y to the reference line 150.

[0063] The performance evaluation device 2000 may treat the wrist height as the height in the vertical direction in the real world, just as it does for the finger joint height. Specifically, the performance evaluation device 2000 identifies the wrist position and reference plane in the world coordinate system from the wrist position and reference plane in the image coordinate system. The distance from the wrist position to the reference plane in the world coordinate system is then treated as the wrist height.

[0064] <<Hand top height>> "Height of the hand top surface" is an evaluation item that indicates the position of the hand top surface in the height direction. For example, the height of the hand top surface is determined by the positional relationship between the third joint point of the middle finger (see FIG. 5) and a reference plane. This reference plane can be the same as the reference plane that determines the height of the finger joints. Note that the height of the hand top surface is preferably calculated for a hand in which at least one finger is typing. However, the height of the hand top surface may also be calculated for a hand in which none of the fingers are typing.

[0065] Fig. 9 is a diagram illustrating a method for calculating the height of the hand upper surface. Here, the hand 140, the keyboard, and the reference line 150 shown in Fig. 9 are the same as those shown in Fig. 8. In Fig. 9, the position of the upper surface of the hand 140 is represented by point W, which is the third joint point of the middle finger of the hand 140.

[0066] 9(a) and (b) show different methods for calculating the height of the hand upper surface. In FIG. 9(a), the height of the hand upper surface is determined by the distance of a line segment WK. Point K is the intersection point of a straight line drawn vertically downward from the hand upper surface W to the video frame 22 and the reference line 150. On the other hand, in FIG. 9(b), the height of the hand upper surface is determined by the distance of a line segment WL. Point L is the foot of a perpendicular line drawn from the hand upper surface W to the reference line 150.

[0067] The performance evaluation device 2000 may treat the height of the hand top surface as the height in the vertical direction in the real world, just as it does for the height of the finger joints and wrist. Specifically, the performance evaluation device 2000 identifies the position of the hand top surface and the reference plane in the world coordinate system from the position of the hand top surface and the reference plane in the image coordinate system. Then, the distance from the position of the hand top surface in the world coordinate system to the reference plane is treated as the height of the hand top surface.

[0068] <<Wrist angle>> "Wrist angle" is an evaluation item that indicates the degree of bending of the wrist. The wrist angle can be determined, for example, by the size of the angle formed by a line representing the posture of the arm and a line representing the posture of the top surface of the hand. Note that it is preferable that the wrist angle is calculated for a hand in a state in which at least one finger is typing. However, the wrist angle may also be calculated for a hand in a state in which none of the fingers is typing.

[0069] Fig. 10 is a diagram illustrating a method for calculating the wrist angle. For ease of explanation, Fig. 10 shows only the image area around the hand 140 in the video frame 22. The wrist angle is represented by the magnitude β of the angle formed by lines 170 and 180. Line 170 is a line representing the posture of the arm. On the other hand, line 180 is a line representing the posture of the upper surface of the hand.

[0070] For example, the straight line 170 can be obtained by detecting an edge representing the upper surface of the arm from the video frame 22, and connecting any two points on the edge, or performing straight line fitting using three or more points on the edge. Similarly, the straight line 180 can be obtained by detecting an edge representing the upper surface of the hand from the video frame 22, and connecting any two points on the edge, or performing straight line fitting using three or more points on the edge.

[0071] The angle of the wrist may also be an angle in the world coordinate system. In this case, for example, the performance evaluation device 2000 calculates a line representing the posture of the arm and a line representing the posture of the upper surface of the hand in the world coordinate system, and calculates the angle between these lines as the angle of the wrist. In addition, in the world coordinate system, the posture of the arm and the posture of the upper surface of the hand may be represented by planes instead of lines. In this case, the angle between these two planes can be treated as the angle of the wrist.

[0072] <<Keystroke Depth>> "Keystroke depth" is an evaluation item that indicates the degree to which a key is pressed when playing. Keystroke depth can be expressed as the relative position of a key when pressed compared to when it is normally pressed.

[0073] 11 is a diagram illustrating a method for calculating the depth of a keystroke. For ease of explanation, only the image area of ​​the video frame 22 surrounding the key 190 is shown in FIG.

[0074] Key 190 is a key that is being pressed. Line segment 200 represents the long side of the top surface of key 190 that is farther from camera 10 in the normal state (unpressed state). Meanwhile, line segment 210 represents the long side of the top surface of key 190 that is farther from camera 10 in the pressed state. Note that line segment 200 may be the long side of the top surface of the key next to pressed key 190 that is farther from camera 10.

[0075] 11(a) and (b) illustrate different calculation methods for the keystroke depth. In FIG. 11(a), the keystroke depth is represented by the angle between line segment 200 and line segment 210. Meanwhile, in FIG. 11(b), the keystroke depth is represented by the length of line IJ. Point I is the end point of line segment 200 that is closer to the body of user 60. Point J is the end point of line segment 210 that is closer to the body of user 60.

[0076] Note that values ​​in the world coordinate system may also be used for the keystroke depth. For example, the performance evaluation device 2000 converts the line segments 200 and 210 in the image coordinate system into line segments in the world coordinate system, and uses the size of the angle between these line segments and the distance between their end points as item values ​​for the keystroke depth. Furthermore, the points used are not limited to the end points, and any point on the line segments 200 and 210 may be used based on the similarity relationship.

[0077] <Examples of Usage of the Performance Evaluation Device 2000> To facilitate understanding of the performance evaluation device 2000, an example of how the performance evaluation device 2000 is used will be described. Note that the usage described here is merely an example, and the specific usage of the performance evaluation device 2000 is not limited to the example described here. For example, when dealing with only evaluation items whose item values ​​can be calculated from performance data, the camera 10 does not need to be used.

[0078] Fig. 12 is a diagram illustrating an example of a usage mode of the performance evaluation device 2000. In Fig. 12(a), the performance evaluation device 2000 is realized by a mobile terminal (user terminal 62) operated by a user 60. For example, it is conceivable that the performance evaluation device 2000 can be realized by the user terminal 62 by installing a program for realizing each of the functional components of the performance evaluation device 2000 described above into the user terminal 62.

[0079] The user terminal 62 acquires video data 20 from the camera 10 that captures the user 60 playing the target instrument 70 from the left, and from the camera 10 that captures the user 60 playing the target instrument 70 from the right. The user terminal 62 also acquires performance data 72 from the target instrument 70. As described above, the performance data 72 is MIDI (registered trademark) data or the like. The user terminal 62 then generates user performance information 50 by analyzing the acquired video data 20 and performance data 72, and generates evaluation information 30 by comparing the user performance information 50 with reference information 40.

[0080] On the other hand, in FIG. 12(b), the performance evaluation device 2000 is realized by a server machine 80 that operates upon receiving a request from a user terminal 62. In this usage mode, the user terminal 62 is used as an interface for using the performance evaluation device 2000. Specifically, the user terminal 62 acquires video data 20 from two cameras 10. The user terminal 62 also acquires performance data 72 from a target musical instrument 70. The video data 20 and the performance data 72 are then transmitted from the user terminal 62 to the server machine 80. For example, the video data 20 and the performance data 72 are attached to a request for an evaluation of the performance of the user 60. The server machine 80 analyzes the acquired video data 20 and performance data 72 to generate user performance information 50, and generates evaluation information 30 by comparing the user performance information 50 with the reference information 40. The server machine 80 then transmits a response including the evaluation information 30 to the user terminal 62.

[0081] <Processing flow> 13 is a flowchart illustrating the flow of processing executed by the performance evaluation device 2000 of embodiment 1. The tempo information acquisition unit 2020 acquires the tempo information 100 (S102). The evaluation unit 2040 acquires the reference information 40 corresponding to the designated tempo indicated in the tempo information 100 (S104). The evaluation unit 2040 acquires the user performance information 50 (S106). The evaluation unit 2040 generates the evaluation information 30 using the reference information 40 and the user performance information 50 (S108).

[0082] The process flow executed by the performance evaluation device 2000 is not limited to that shown in Fig. 13. For example, the acquisition of the user performance information 50 (S106) may be executed before the acquisition of the tempo information 100 (S102) and the acquisition of the reference information 40 (S104). Furthermore, these processes may be executed in parallel.

[0083] <Getting tempo information 100: S102> The tempo information acquisition unit 2020 acquires tempo information 100. For example, the tempo information 100 is acquired in response to a user input. That is, the tempo information acquisition unit 2020 acquires tempo information 100 that can identify a tempo specified by a user input.

[0084] In this case, for example, the tempo information acquisition unit 2020 causes a screen for inputting a designated tempo (hereinafter, a tempo designation screen) to be displayed on a display device of a terminal that can be operated by the user. For example, this display device is displayed on a touch panel display provided on a computer (such as the user terminal 62 or the target instrument 70) on which an application that realizes the functions of the performance evaluation device 2000 is installed. Note that the user who designates the designated tempo may be any person and does not have to be the user 60 who is the performer. The tempo information acquisition unit 2020 acquires information indicating the result of the user's input on the screen as tempo information 100.

[0085] Fig. 14 is a diagram illustrating a tempo designation screen 90. In the example of Fig. 14(a), the user inputs a designated tempo by inputting a specific numerical value. In this case, tempo information 100 indicates the specific numerical value input by the user (i.e., the specific numerical value representing the designated tempo). Therefore, the tempo information acquisition unit 2020 identifies the value indicated in the tempo information 100 as the designated tempo.

[0086] On the other hand, in the example of FIG. 14(b), multiple options for the designated tempo are displayed, and the designated tempo is input by selecting one of them. Specifically, in FIG. 14(b), one of three tempos, "normal," "slightly slow," and "slow," can be selected. In this case, tempo information 100 indicates the option selected by the user as information that can identify the designated tempo. The tempo information acquisition unit 2020 identifies the designated tempo based on the information indicated in the tempo information 100.

[0087] For example, the tempo information acquisition unit 2020 identifies a normal tempo corresponding to the target song, and identifies a designated tempo based on the normal tempo and the options shown in the tempo information 100. The normal tempo corresponding to the target song is stored in advance in a storage device accessible from the performance evaluation device 2000, for example, in association with identification information of the target song. In addition, for relative tempos such as "slightly slow," a ratio to the normal tempo is determined in advance.

[0088] For example, in FIG. 14(b), assume that "slightly slow" is 2 / 3 the normal tempo, and "slow" is 1 / 2 the normal tempo. Also assume that the normal tempo of the target song is 120 BPM (Beats Per Minute). In this case, the tempos corresponding to "slightly slow" and "slow" are 80 BPM and 60 BPM, respectively. Therefore, the tempo information acquisition unit 2020 specifies 120 BPM as the designated tempo when "normal" is selected, specifies 80 BPM as the designated tempo when "slightly slow" is selected, and specifies 60 BPM as the designated tempo when "slow" is selected.

[0089] The tempo information acquisition unit 2020 may include specific tempo values ​​(for example, the above-mentioned 120 BPM, 80 BPM, and 60 BPM) corresponding to each option in the tempo designation screen 90 of Fig. 14(b). In this case, the tempo information acquisition unit 2020 specifies the normal tempo of the target music piece before generating the tempo designation screen 90. The tempo information acquisition unit 2020 also specifies the tempo corresponding to each option based on the normal tempo of the target music piece.

[0090] <Acquisition of standard information 40: S104> The evaluation unit 2040 acquires the reference information 40 (S104). The reference information 40 is information indicating reference item values ​​for each evaluation item for a given piece of music. The reference item values ​​are ideal item values ​​obtained, for example, by analyzing the performance of an advanced player such as an instructor. Hereinafter, the person who performs to generate the reference information 40 is referred to as an exemplary performer. The reference information 40 is generated in advance for each piece of music that can be evaluated, and is stored in a storage device accessible from the performance evaluation device 2000.

[0091] The reference information 40 is time-series data showing the item values ​​of each evaluation item in a time series. More specifically, the reference information 40 shows, for each of a plurality of performance time points, a combination of "performance time point, item value of each evaluation item for the performance at that performance time point." Each performance time point shown in the reference information 40 is, for example, a time point at which at least one sound is played. That is, in this case, the reference information 40 shows, for each time point at which at least one sound is played, a standard for how the performance should be performed at that time point.

[0092] The starting point of the performance time (the point when the value is 0) is, for example, the start point of a piece of music. The starting point of a piece of music is, for example, the point when the piece of music is first played (such as a keystroke). Hereinafter, unless otherwise specified, the starting point of the performance time indicated in the reference information 40 is assumed to be the start point of the piece of music.

[0093] 15 is a diagram illustrating an example of the reference information 40 in the form of a table. The reference information 40 has item information 44 for each performance time point 42. The item information 44 indicates, for each evaluation item, a pair of an evaluation item 46 and an item value 48 for the performance at the corresponding performance time point. The evaluation item 46 indicates identification information (e.g., the name of the evaluation item) that specifies the evaluation item. The item value 48 indicates the item value of the corresponding evaluation item.

[0094] Here, the performance time 42 corresponding to each item information 44 differs depending on the tempo of the performance. For example, a sound played at time t1 at a tempo of 120 BPM will be played at time 2*t1 at a tempo of 60 BPM. Therefore, even for the same piece of music, the reference information 40 to be used will differ for each designated tempo.

[0095] Therefore, for example, multiple pieces of reference information 40 corresponding to different tempos are prepared in advance for each target song. In this case, the evaluation unit 2040 acquires the reference information 40 corresponding to the designated tempo from the multiple pieces of reference information 40 prepared for the target song. For example, suppose that there are three tempos that can be designated when evaluating the performance of the song M1: 120 BPM, 80 BPM, and 60 BPM. Also, suppose that the designated tempo identified based on the tempo information 100 is 80 BPM. In this case, the evaluation unit 2040 acquires the reference information 40 corresponding to the tempo of 80 BPM from the three pieces of reference information 40 prepared for the song M1.

[0096] Alternatively, for example, one piece of reference information 40 may be prepared for each target song, and the value of the performance time point 42 may be corrected according to the designated tempo. For example, reference information 40 for a case where the target song is played at a normal tempo is prepared in advance for each target song. In this case, the reference information 40 further includes information indicating the normal tempo. The evaluation unit 2040 converts the reference information 40 corresponding to the normal tempo into the reference information 40 corresponding to the designated tempo by correcting each performance time point 42 indicated by the reference information 40 acquired for the target song based on the ratio between the designated tempo identified using the tempo information 100 and the normal tempo indicated by the reference information 40. In this way, the reference information 40 corresponding to the designated tempo can be acquired.

[0097] For example, as mentioned above, a sound played at time t1 at a tempo of 120 BPM will be played at time 2*t1 at a tempo of 60 BPM. In other words, if we generalize this by expressing the standard tempo as r1 [BPM] and the specified tempo as r2 [BPM], the reference information 40 corresponding to tempo r1 can be converted to reference information 40 corresponding to tempo r2 by multiplying each performance time indicated by the reference information 40 corresponding to tempo r1 by r1 / r2.

[0098] Note that even when obtaining the reference information 40 corresponding to the designated tempo by correcting the reference information 40 in this way, multiple pieces of reference information 40 each having a different tempo may be prepared for each target song. In this case, for example, if the reference information 40 corresponding to the designated tempo is prepared in advance, the evaluation unit 2040 uses the reference information 40. On the other hand, if the reference information 40 corresponding to the designated tempo is not prepared in advance, the evaluation unit 2040 identifies the reference information 40 corresponding to the tempo closest to the designated tempo from the multiple pieces of reference information 40, and performs the above-mentioned correction on the reference information 40 to generate the reference information 40 corresponding to the designated tempo. In this way, by correcting the reference information 40 corresponding to the tempo closest to the designated tempo, it is possible to minimize errors that may occur due to the correction.

[0099] For example, for the song M1, two pieces of reference information 40 corresponding to 60 BPM and 120 BPM are prepared in advance. Also, assume that a tempo of 100 BPM is specified. In this case, the evaluation unit 2040 generates the reference information 40 corresponding to 100 BPM by correcting the reference information 40 corresponding to 120 BPM, which is closer to the specified tempo. Note that it is preferable to specify the closeness of the tempos by the ratio of the contrasting tempos. For example, in the above example, the closeness between 60 BPM and 100 BPM is 5 / 3 (=100 / 60), and the closeness between 100 BPM and 120 BPM is 6 / 5 (=120 / 100).

[0100] <<Identifying target songs>> Here, in order to obtain the reference information 40 about the target piece of music, it is necessary to identify the target piece of music. Also, as described above, when the tempo specified by the user is a relative tempo, it is necessary to know the normal tempo of the target piece of music, so it is necessary to identify the target piece of music. Therefore, the performance evaluation device 2000 identifies the target piece of music.

[0101] There are various methods for the performance evaluation device 2000 to specify the target piece of music. For example, the performance evaluation device 2000 accepts a user input specifying the target piece of music. Note that the user who specifies the target piece of music may be any person, and does not have to be the user 60 who is the performer.

[0102] 16 is a diagram illustrating a designation screen (song designation screen 300) for designating a target song. Like the tempo designation screen 90, the song designation screen 300 is displayed on a display device of a terminal that can be operated by the user. The song designation screen 300 includes a search area 310, a song selection area 320, and an evaluation range designation area 330. The search area 310 is an input interface for searching for a target song using search words such as song title or attributes (genre, etc.).

[0103] The search for the target song is not limited to a search using a search word. For example, the evaluation unit 2040 may acquire audio data of the target song and search a database of songs using the audio data to identify the target song.

[0104] The search results are displayed in the song selection area 320. The user specifies the target song by selecting one of the songs displayed in the song selection area 320. In the example of FIG. 16, the song "Song F" is selected. Note that before the search is performed, candidate songs may or may not be displayed in the song selection area 320. In the former case, for example, the song selection area 320 displays all songs that can be selected as target songs. Alternatively, for example, the song selection area 320 may display a list of songs that have been selected as target songs (i.e., the history of songs to date).

[0105] The evaluation range designation area 330 is used when it is desired to evaluate only a portion of a song. In this example, both the start position and the end position can be designated. Hereinafter, the portion of a song selected as the evaluation target will be referred to as the "evaluation range."

[0106] There are various specific methods for specifying the evaluation range. For example, a song is divided into a plurality of sections (such as an intro and a chorus) in advance, and one or more sections can be specified as the evaluation range by user input. When the evaluation range can be specified on a section-by-section basis in this way, the reference information 40 includes information that can identify which section each piece of item information 44 is included in. For example, the reference information 40 further includes information such as "the name of the section, the start time of the section, and the end time of the section" for each of all sections included in the corresponding song. When the evaluation range is specified on a section-by-section basis, the evaluation unit 2040 uses, for evaluation, each piece of item information 44 included in the reference information 40 of the target song that is included in the range from the start time of the first section of the evaluation range to the end time of the last section of the evaluation range.

[0107] The method of specifying the evaluation range is not limited to the method of specifying a section. For example, the evaluation range may be specified in units of bars. When the evaluation range is specified in units of bars in this way, the reference information 40 includes information that can identify which bar each item information 44 is included in. For example, the reference information 40 further includes information such as "bar number, start time of the bar, and end time of the bar" for each of all bars included in the corresponding music piece. When the evaluation range is specified in units of bars, the evaluation unit 2040 uses, for evaluation, each item information 44 included in the reference information 40 of the target music piece that is included in the range from the start time of the first bar of the evaluation range to the end time of the last bar of the evaluation range, among the item information 44 included in the reference information 40 of the target music piece.

[0108] However, in the case of a piece of music in which the length of each bar is constant, it is possible to convert a bar number into time if the length of each bar is known. Therefore, when dealing with a piece of music in which the length of each bar is constant, the length of one bar may be included in the reference information 40 instead of the time range of each bar. In this case, the evaluation unit 2040 calculates the start time of the first bar in the evaluation range and the end time of the last bar in the evaluation range using the length of one bar indicated in the reference information 40. The evaluation unit 2040 then identifies the item information 44 in which the performance time 42 is included in the range from the start time to the end time as the item information 44 included in the evaluation range, and uses it for evaluation.

[0109] In this case where only a portion of the target music piece is to be evaluated, the prepared reference information 40 is corrected to obtain the reference information 40 corresponding to the specified tempo. In this case, the evaluation unit 2040 does not correct the entire reference information 40, but only the portion corresponding to the evaluation range.

[0110] The music selection screen 300 may further include an input interface that allows various conditions related to the evaluation to be specified. For example, the evaluation conditions include the age group of the user 60. The age group of the user 60 is, for example, either "adult" or "child". In addition, more detailed designation such as "infant" or "elementary school student" may be possible. When the age group of the user 60 can be designated in this way, it is preferable to prepare the reference information 40 for each age group. This is because the size of the hand varies depending on the age group of the performer, and this difference affects the evaluation value.

[0111] Another example of an evaluation condition is the type of instrument. For example, the types of instruments may be "acoustic piano," "electronic piano," "pipe organ," "electronic organ," or "synthesizer." When the type of instrument can be specified in this way, it is preferable to prepare reference information 40 for each type of instrument. This is because the hardness of the keys (the force required to strike the keys) differs depending on the type of instrument, and this difference affects the evaluation value.

[0112] The input interface for specifying the evaluation conditions may be provided as a screen separate from the music selection screen 300.

[0113] Also, the music selection screen 300 and the tempo selection screen 90 may be integrated into one screen. That is, the music selection screen 300 may include an area for specifying the tempo.

[0114] <<How to generate Reference Information 40>> For example, the reference information 40 can be generated by analyzing the video data 20 in which a performance by a model performer is recorded, or the performance data 72. Here, a device that generates the reference information 40 is called a reference information generating device. The reference information generating device may be realized by the same computer as the performance evaluation device 2000, or may be realized by a different computer.

[0115] When performing a model performance for generating reference information 40, the model performer inputs information specifying the tempo of the performance. The reference information generating device includes information indicating the specified tempo in the reference information 40 that it generates.

[0116] When generating reference information 40 for evaluation items that can be evaluated using visual information such as the shape of hands and fingers, the reference information generating device generates the reference information 40 using video data 20. Specifically, the reference information generating device acquires video data 20 in which a performance performed by a model performer is recorded, and extracts each of the video frames 22 in which a keystroke is performed from the video frames 22 included in the video data 20. Furthermore, the reference information generating device analyzes the extracted video frames 22 to calculate item values ​​for each evaluation item. Then, for each extracted video frame 22, the reference information generating device generates the reference information 40 by associating the performance time point corresponding to the video frame 22 with the item information obtained from the video frame 22 (multiple pairs of "evaluation item, item value").

[0117] In order to calculate the item value of each evaluation item from the video frame 22, the reference information generating device performs image analysis on the video frame 22. Specifically, the reference information generating device extracts information required to calculate the item value of each evaluation item from the video frame 22. The reference information generating device further calculates the item value of each evaluation item using the extracted information. Then, for each video frame 22, the reference information generating device generates user performance information 50 indicating a combination of the performance time point corresponding to that video frame 22 and the item value calculated for each evaluation item using that video frame 22.

[0118] The item value of each evaluation item can be calculated from the video frame 22 using the various methods described above with reference to Figures 6 to 11. Information required for calculating the item value includes, for example, the position of the knuckles of each finger, the position of the wrist of each hand, the various reference lines described above, a line representing the posture of the arm, or a line representing the posture of the upper surface of the hand.

[0119] For example, when calculating the height of a finger joint, the reference information generating device detects each finger joint from the video frame 22 and identifies its position. The reference information generating device also detects the key on which each finger is placed and identifies the above-mentioned reference line based on that key. Then, the reference information generating device calculates the height of the finger joint based on the identified finger joint position and reference line.

[0120] When calculating the magnitude of the angle between joint lines, the reference information generating device detects and identifies the positions of each finger joint from the video frame 22. Then, for a plurality of joint lines that can be determined from the positions of these finger joints, the reference information generating device calculates the magnitude of the angle between each pair of two joint lines that share a common finger joint.

[0121] When calculating the wrist height, the reference information generating device detects the joint representing the wrist position from the video frame 22 and identifies the position. The reference information generating device also identifies the key on which the hand is placed to determine the reference line, thereby identifying the reference line. The reference information generating device then calculates the wrist height based on the identified wrist position and reference line.

[0122] When calculating the height of the hand top surface, the reference information generating device identifies the position of the hand top surface (for example, the position of the third joint point of the middle finger) from the video frame 22. The reference information generating device also identifies a reference line. The method of identifying the reference line can be the same as in the case of calculating the wrist height. Then, the reference information generating device calculates the height of the hand top surface based on the identified position of the hand top surface and the reference line.

[0123] When calculating the angle of the wrist, the reference information generating device detects the hand and arm from the video frame 22 and identifies a straight line representing the posture of the upper surface of the hand and a straight line representing the posture of the arm. The reference information generating device then calculates the angle of the wrist using these straight lines.

[0124] When calculating the depth of keystrokes, the reference information generating device detects the key being pressed from the video frame 22. Then, for each detected key, the reference information generating device calculates a line segment representing the long side of the top surface of the key that is farther from the camera 10 in both the pressed state and the normal state. Then, the reference information generating device calculates the depth of keystrokes of each key using these line segments. Note that information about a key in the normal state can be obtained from the line segment of the top side of the key in the video frame 22 when the key is not pressed, or from the line segment of the top side of the key next to the key that is farther from the camera 10 in the video frame 22 when the key is pressed.

[0125] As described above, the world coordinate system may be used to calculate each item value. In this case, the reference information generating device converts information (such as the position of the finger joint) detected from the video frame 22 into information on the world coordinate system, as described above, and then calculates the item value.

[0126] The fingering is also identified using the video data 20. Specifically, the reference information generating device detects the pressed key and the finger that pressed the key from the video frame 22, and generates reference information 40 based on the detection result. A performance time point 42 indicates the performance time point corresponding to the video frame 22. An evaluation item 46 indicates the name of the evaluation item "fingering" and the identification information of the note corresponding to the detected key. An item value 48 indicates the identification information of the detected finger.

[0127] In order to be able to identify the pressed key from the video frame 22, for example, information is created in advance that represents the correspondence between the identification information of the key and the position of the image area representing that key on the video frame 22. By referring to this information, it is possible to identify the identification information of the pressed key from the position of the pressed key on the video frame 22.

[0128] Here, in order to generate the reference information 40 using the video frame 22, it is necessary to identify the performance time point corresponding to that video frame 22. For example, as described above, the performance time point is expressed as a relative time point based on the time point at which the first performance (key striking) is performed in the target music piece. In this case, for example, the reference information generating device sets the generation time point of the video frame 22 representing the first performance among the video frames 22 included in the acquired video data 20 as the reference time point. For the other video frames 22, the reference information generating device converts the generation time point of the video frame 22 for which the performance time point is to be identified to the performance time point based on the difference between the generation time point of the video frame 22 for which the performance time point is to be identified and the generation time point of the video frame 22 representing the reference time point, and the frame rate of the camera 10.

[0129] The method of identifying the time when the target piece of music is first played is not limited to using the time when the video frame 22 representing the first performance is generated. For example, when the model performer plays the target piece of music, a metronome sound may be output so that the performance start timing and the performance tempo can be easily grasped. In this case, the reference information generating device may identify the time when the target piece of music is played based on the metronome sound.

[0130] For example, if the target piece of music is in 4 / 4 time, the model performer starts playing after the metronome has sounded four times. In this case, if the time when the fourth metronome sound is ta and the time interval between the metronome sounds is i, the time ta+i can be specified as the time when the first performance is performed.

[0131] The metronome sound may be output from the reference information generating device, or may be output from something else (for example, a mechanical metronome). In the latter case, the reference information generating device uses audio data in which the metronome sound is recorded to identify the time when the metronome sound is generated, the time interval of the metronome sound, etc.

[0132] When generating reference information 40 for evaluation items related to sound characteristics, for example, the reference information generating device uses performance data 72 in which a performance by a model performer is recorded. The performance data 72 includes information for each performed sound that can identify the time of performance, sound identification information, and various sound characteristics (strength, duration, etc.). In MIDI (registered trademark) terminology, sound identification information and sound strength are called "note number" and "velocity," respectively.

[0133] When generating reference information 40 using performance data 72, for example, the reference information generating device can generate a record of reference information 40 regarding sound intensity by extracting, for each played note, information such as the playing time, note number, and velocity of that note from the acquired performance data 72. Similarly, the reference information generating device can generate a record of reference information 40 regarding sound duration by extracting, for each played note, information such as the playing time, note number, and duration of that note from the acquired performance data 72.

[0134] A more specific explanation will be given of a case where MIDI (registered trademark) data is used as performance data. In MIDI (registered trademark) data, the output and stop of a specific sound are represented by a note-on message, which indicates that the sound is played, and a note-off message, which indicates that the sound is stopped. The strength of the sound can be specified by the velocity value indicated in the note-on message. Meanwhile, the length of the sound can be specified by the difference between the time when the note-on message is played and the time when the note-off message is played.

[0135] Alternatively, for example, the reference information generating device may obtain video data 20 in which a performance by a model performer is recorded, and generate reference information 40 regarding sound characteristics using the video data 20. Here, one of the external characteristics of performance that affects the sound intensity is the keystroke speed. Specifically, the faster the keystroke speed, the stronger the sound intensity. Thus, the basic information generating device analyzes the video data 20 to identify the "keystroke time (play time), identification information of the keystroke sound, and keystroke speed" for each keystroke performed by the model performer. This makes it possible to generate reference information 40 regarding the intensity of each sound played. The play time may be represented as the start time of the keystroke motion, or may be represented as the end time of the keystroke motion.

[0136] The speed of a keystroke when a certain key is pressed can be expressed, for example, as the inverse of the difference between the start time ts of the keystroke and the end time te of the keystroke, 1 / (te-ts). The start time of the keystroke can be expressed as the generation time of the video frame 22 representing the start of the keystroke (the video frame 22 when the key starts to be lowered). Similarly, the end time of the keystroke can be expressed as the generation time of the video frame 22 representing the end of the keystroke (the video frame 22 when the key is lowered to the bottom).

[0137] Another example of an external characteristic of a performance that affects the strength of a sound is the depth of keystroke. Specifically, the deeper the keystroke, the louder the sound. In addition, some types of musical instruments do not produce sound unless the key is struck to a certain depth. The basic information generating device analyzes the video data 20 to identify the "keystroke time (play time), identification information of the struck sound, and the depth of keystroke" for each keystroke performed by the model performer. This makes it possible to generate reference information 40 regarding the strength of each sound played. The play time may be represented as the start time of the keystroke motion or the end time of the keystroke motion. In addition, the depth of keystroke can be represented by using the angle of the reference line and the length of the line segment between the reference points, as described in FIG. 11 of the first embodiment.

[0138] It should be noted that the reference information 40 regarding the sound intensity may indicate only one of the keystroke speed and the keystroke depth, or may indicate both.

[0139] When the length of a sound is specified using the video data 20, it can be specified, for example, as the difference between the time when the key is pressed and the time when the key is released. The reference information generating device analyzes the video data 20 to specify the "time when the key is pressed (time when the key is played), the identification information of the pressed sound, and the time when the key is released" for each keystroke performed by the model performer. This makes it possible to generate reference information 40 regarding the length of each played sound.

[0140] Information obtained from the performances of each of a plurality of model performers may be used to generate the reference information 40. For example, the reference information generating device generates reference information 40 for each of a plurality of model performers in the above-described manner, and then performs statistical processing on the generated plurality of pieces of reference information 40 to generate reference information 40 to be used in evaluating the performances. For example, it is conceivable to use statistics (such as average values) of item values ​​obtained for each of a plurality of people.

[0141] <Acquisition of user performance information 50: S106> The evaluation unit 2040 acquires the user performance information 50 (S106). The user performance information 50 indicates, for a performance of the target piece of music performed by the user 60, a combination of evaluation items and item values ​​at each of a plurality of performance times in association with each of the performance times.

[0142] FIG. 17 is a diagram illustrating an example of user performance information 50 in the form of a table. The configuration of the user performance information 50 shown in FIG. 17 is the same as the configuration of the reference information 40 shown in FIG. 15. Specifically, the user performance information 50 has item information 54 for each performance time point 52. The item information 54 indicates, for each evaluation item, a pair of an evaluation item 56 and an item value 58 for the performance at the corresponding performance time point. The evaluation item 56 indicates identification information (e.g., the name of the evaluation item) that specifies the evaluation item. The item value 58 indicates the item value of the corresponding evaluation item.

[0143] The user performance information 50 may be generated by the performance evaluation device 2000 or by a device other than the performance evaluation device 2000. In the former case, for example, the performance evaluation device 2000 acquires video data 20 generated by capturing a performance by the user 60 with the camera 10 and performance data 72 representing the performance given by the user 60, and analyzes these to generate the user performance information 50. Here, the method of generating the user performance information 50 using the video data 20 is the same as the method of generating the reference information 40 using the video data 20. Moreover, the method of generating the user performance information 50 using the performance data 72 is the same as the method of generating the reference information 40 using the performance data 72.

[0144] As described above, in generating reference information using the video data 20, the sound of a metronome may be used to facilitate understanding of the timing of the start of performance and the tempo of the performance. In this regard, the sound of a metronome may also be used when the user 60 plays the target song. In this case, the evaluation unit 2040 can use the sound of the metronome to identify the time when the user 60 first performed the song (i.e., time 0 in the user performance information 50).

[0145] For example, if the target song is in 4 / 4 time, the user 60 starts playing after the metronome sounds four times. This allows the user 60 to easily grasp the correct timing to start playing. In particular, if the evaluation range starts in the middle of the target song, it may be difficult to start playing compared to the case where the song is played from the beginning. In this regard, by starting playing after listening to the metronome sound, it becomes possible to start playing more naturally.

[0146] The metronome sound may be set to continue to be output while the user 60 is performing. This makes it easier to maintain the correct tempo while performing. However, since there may be cases where the tempo of the performance needs to be evaluated, it may be possible to set the metronome sound not to be output during performance.

[0147] The sound of the metronome may be output from the performance evaluation device 2000, or may be output from something else (for example, a mechanical metronome). In the former case, the performance evaluation device 2000 sets the tempo of the metronome to the designated tempo indicated in the tempo information 100.

[0148] Various methods can be used for the evaluation unit 2040 to acquire the video data 20. For example, when the performance evaluation device 2000 is realized by a computer other than the camera 10, the evaluation unit 2040 acquires the video data 20 by receiving the video data 20 transmitted from the camera 10 or by accessing a storage device in which the video data 20 is stored and reading the video data 20. Note that the storage device in which the video data 20 is stored may be provided either inside or outside the camera 10. When the performance evaluation device 2000 is realized by the camera 10, the evaluation unit 2040 acquires the video data 20 by reading the video data 20 stored inside the performance evaluation device 2000 (for example, the storage device 1080).

[0149] Various methods can be used to acquire the performance data 72. For example, the evaluation unit 2040 acquires the performance data 72 by receiving the performance data 72 transmitted by a device (such as the target musical instrument 70) that generated the performance data 72, or by accessing a storage device in which the performance data 72 is stored and reading the performance data 72.

[0150] When the user performance information 50 is generated by a device other than the performance evaluation device 2000, the evaluation unit 2040 acquires the user performance information 50 generated by the other device. Various methods can be used for acquiring the user performance information 50. For example, the evaluation unit 2040 acquires the user performance information 50 by receiving the user performance information 50 transmitted from the device that generated the user performance information 50, or by accessing a storage device in which the user performance information 50 is stored and reading the user performance information 50.

[0151] <Generation of evaluation information 30: S108> The evaluation unit 2040 generates the evaluation information 30 using the reference information 40 and the user performance information 50. Specifically, the evaluation unit 2040 associates the item information 44 included in the reference information 40 with the item information 54 included in the user performance information 50, and compares the item information 44 and the item information 54 that correspond to each other. The corresponding item information 44 and the item information 54 are pieces of information that correspond to each other at the same or close performance time points, and should represent the same performance action (keystroke, etc.). More specifically, for example, the evaluation unit 2040 calculates an evaluation score that indicates the degree of agreement of the item values ​​for each evaluation item for the corresponding item information 44 and the item information 54. Then, the evaluation unit 2040 generates the evaluation information 30 using the calculated evaluation score. Note that a method for calculating the evaluation score and a method for generating the evaluation information 30 using the evaluation score will be described later.

[0152] <<Process flow>> FIG. 18 is a flow chart illustrating a process flow for calculating an evaluation score for each item information 44. The loop process L1 is executed for each item information 44 included in the reference information 40. In S202, the evaluation unit 2040 judges whether the loop process L1 has been executed for all the item information 44. If the loop process L1 has already been executed for all the item information 44, the process of FIG. 18 ends. On the other hand, if there is one or more item information 44 that has not yet been targeted by the loop process L1, the evaluation unit 2040 selects one item information 44 from among them. For example, the evaluation unit 2040 selects the item information 44 as the target of the loop process L1 in order from the item information 44 with the earliest corresponding performance time point 42. The item information 44 selected in S202 is denoted as item information 44-i.

[0153] The evaluation unit 2040 identifies the item information 54 corresponding to the item information 44-i from among the item information 54 included in the user performance information 50 (S204). For example, the evaluation unit 2040 identifies the performance time point 52 indicated in the user performance information 50 that is closest to the performance time point 42 corresponding to the item information 44-i. Then, the evaluation unit 2040 identifies the item information 54 corresponding to the identified performance time point 52 as the item information 54 corresponding to the item information 44-i.

[0154] The evaluation unit 2040 calculates an evaluation score for each evaluation item by comparing the identified item information 54 with the item information 44-i (S206). Since S208 is the end of the loop process L1, the process in FIG.

[0155] If there is no performance time 52 whose difference from the performance time 42 corresponding to the item information 44-i is equal to or less than a predetermined value (in other words, if the difference between the performance time 42 and the performance time 52 closest to the performance time 42 is greater than a predetermined value), the evaluation unit 2040 may determine that there is no item information 54 corresponding to the item information 44-i and not execute S206 for the item information 44-i. This may occur, for example, when a sound is missing in the performance by the user 60.

[0156] <<How the evaluation score is calculated>> The evaluation score represents the degree of agreement between item value 58 shown in item information 54 and item value 48 shown in item information 44 for a target evaluation item. For example, an evaluation formula for calculating an evaluation score is prepared in advance for each evaluation item. The value indicated by item value 48 in item information 44 and the value indicated by item value 58 in item information 54 for a target evaluation item are input into the evaluation formula. The evaluation formula then outputs an evaluation score by performing a predetermined calculation on the input values.

[0157] For example, the evaluation formula outputs a value that represents the difference between item value 48 and item value 58, or the ratio of item value 58 to item value 48. The difference or ratio may be standardized.

[0158] Alternatively, for example, the evaluation formula may be configured to output a determination result as to whether or not the item value 58 is included in an allowable range determined based on the item value 48. For example, a range such as "the value indicated by the item value 48 ±20% (0.8 times or more the value indicated by the item value 48, and 1.2 times or less the value indicated by the item value 48)" is defined as the allowable range. The evaluation formula acquires the item value 48 and the item value 58, and determines whether or not the item value 58 is included in the allowable range based on the item value 48. The evaluation formula then outputs a value indicating the determination result (for example, 1 if it is included in the allowable range, 0 if it is not included).

[0159] Alternatively, for example, a number of numerical ranges may be defined based on the item value 48 to evaluate the performance of the user 60 with a rank. For example, assume that a numerical range A of "a value indicated by the item value 48 ±20%", a numerical range B of "a value indicated by the item value 48 ±40% outside the numerical range A", and a numerical range C of "outside the numerical range B" are defined. In this case, when the item value 58 is included in the numerical range A, the evaluation is the highest, when the item value 58 is included in the numerical range B, the evaluation is the second highest, and when the item value 58 is included in the numerical range C, the evaluation is the lowest. Thus, a rank representing such a high evaluation is assigned to each numerical range. For example, a rank represented by a smaller natural number is assigned to a numerical range with a higher evaluation. In the above example, ranks 1, 2, and 3 are assigned to the numerical ranges A to C, respectively.

[0160] When performing an evaluation by rank in this way, the evaluation formula acquires item value 48 and item value 58, and identifies the numerical range that includes item value 58 from among the multiple numerical ranges. The evaluation formula then outputs a rank corresponding to the identified numerical range. For example, in the above example, if item value 58 is included in numerical range B, a rank of "2" is output.

[0161] The evaluation score for fingering indicates the result of the determination as to whether the fingering indicated in the reference information 40 matches the fingering indicated in the user performance information 50. That is, for corresponding item information 44 and item information 54, the evaluation unit 2040 determines whether the finger identification information indicated in item value 48 matches the finger identification information indicated in item value 58, and generates an evaluation score indicating the determination result. For example, the evaluation score indicates "1" if the fingering matches, and indicates "0" if the fingering does not match.

[0162] <<Generation of evaluation information 30 using evaluation scores>> The evaluation unit 2040 uses the evaluation scores to generate evaluation information 30. For example, the evaluation unit 2040 generates evaluation information 30 indicating all the calculated evaluation scores. More specifically, the evaluation unit 2040 generates evaluation information 30 indicating a plurality of pairs of "evaluation item, evaluation score" generated by comparing item information 44 and item information 54 corresponding to each time point at which a key is struck in the target song (performance time point 42).

[0163] Fig. 19 is a diagram illustrating an example of the evaluation information 30 in the form of a table. The table shown in Fig. 19 is denoted as table 400. Table 400 shows a performance time 402 and an evaluation result 404. The evaluation result 404 shows a pair of an evaluation item 406 and an evaluation score 408. The evaluation score 408 shows the evaluation score for the corresponding evaluation item 406 for the performance of the user 60 at the corresponding performance time 402.

[0164] Alternatively, for example, the evaluation unit 2040 may generate the evaluation information 30 by tallying up the evaluation scores in various ways. For example, the evaluation unit 2040 calculates, for each performance time point, a statistical value (such as an average value) of the evaluation scores calculated for the item information 54 corresponding to that performance time point. In this case, a score representing a comprehensive evaluation of the performance at each time point is obtained. Hereinafter, the statistical value of the evaluation score is referred to as a statistical score. When the importance of each evaluation item differs, the statistical score may be a value (for example, a weighted average) that takes into account the importance of each evaluation item.

[0165] Fig. 20 is a diagram illustrating an example of evaluation information 30 indicating a statistical score for each performance time point. The table shown in Fig. 20 is denoted as table 500. Table 500 indicates a pair of a performance time point 502 and a statistical score 504. The statistical score 504 is a statistical value of a plurality of evaluation scores calculated for the corresponding performance time point 502.

[0166] The evaluation scores may be tallied for each evaluation item. That is, the evaluation unit 2040 calculates a statistical value of the evaluation scores calculated for each evaluation item at each of a plurality of performance points. In this case, a comprehensive evaluation of the performance of the user 60 is shown for each evaluation item.

[0167] Fig. 21 is a diagram illustrating evaluation information 30 indicating a statistical score for each evaluation item. The table shown in Fig. 21 is denoted as table 600. Table 600 indicates a pair of evaluation item 602 and statistical score 604. Statistical score 604 is a statistical value of multiple evaluation scores calculated for the corresponding evaluation item 602.

[0168] In the example of FIG. 21, the evaluation scores are tallied for the entire evaluation range. However, the evaluation scores may be tallied for each of a plurality of partial ranges included in the evaluation range. For example, the partial ranges are sections or bars. When the evaluation scores are tallied for each partial range, the reference information 40 includes information that can identify each partial range (for example, information that can identify a section). The evaluation unit 2040 identifies each partial range with reference to the reference information 40, and thereby groups the evaluation scores for each partial range. The evaluation unit 2040 then tallies the evaluation scores for each partial range (for example, calculates a statistical score).

[0169] Fig. 22 is a diagram illustrating evaluation information 30 indicating statistical scores for each subrange and for each evaluation item. The table shown in Fig. 22 is denoted as table 700. Table 700 includes subranges 702 and evaluation results 704. Evaluation results 704 include evaluation items 706 and statistical scores 708. Statistical scores 708 indicate statistical values ​​calculated from evaluation scores for the corresponding evaluation items 706, out of the evaluation scores calculated for the performance time points included in the corresponding subranges 702.

[0170] When the evaluation items can be classified into multiple groups, the evaluation scores may be tallied for each group of evaluation items. For example, evaluation items related to hands can be classified into evaluation items related to the right hand and evaluation items related to the left hand. Furthermore, evaluation scores for evaluation items related to fingers can be classified into evaluation scores for the thumb, index finger, middle finger, ring finger, and little finger.

[0171] Fig. 23 is a diagram illustrating an example of the statistical values ​​of the evaluation scores tallied for each group. The table shown in Fig. 23 is referred to as table 800. Table 800 includes a performance time 802 and an evaluation result 804. The evaluation result 804 includes an item group 806 and an evaluation score 808. The evaluation score 808 indicates the statistical values ​​of the evaluation scores of each evaluation item belonging to the item group 806 for the performance at the corresponding performance time 802.

[0172] Incidentally, even in the case where the evaluation scores are tallied for each group in this way, the statistical scores may be calculated for the entire evaluation range or a partial range, rather than for each performance time point.

[0173] <<Detection of tempo deviation>> The evaluation unit 2040 may detect a deviation in the performance tempo (hereinafter, tempo deviation) by comparing the reference information 40 with the user performance information 50. "A deviation in the performance tempo" means that the order of the notes being played is correct, but the playing time is deviated from the ideal time.

[0174] If the user 60 cannot play the target piece of music at the correct tempo, it can be said that the user 60 has not yet reached a level of proficiency at which detailed evaluations such as evaluation of the shape of the hands and fingers and evaluation of the characteristics of the sound can be performed. Therefore, it is preferable to practice at a slower tempo before performing these evaluations to prevent tempo deviations. Therefore, when the evaluation unit 2040 detects a tempo deviation, it may output a message encouraging the user to practice at a slower tempo.

[0175] The method by which the evaluation unit 2040 detects a tempo deviation is arbitrary. For example, the evaluation unit 2040 divides the time axis into a plurality of blocks, and compares the sounds shown in the reference information 40 and the sounds shown in the user performance information 50 for each block. Here, if the plurality of sounds shown in the reference information 40 and the plurality of sounds shown in the user performance information 50 match in both the type and order of the sounds in a certain block, but the performance time is different, it can be said that a tempo deviation has occurred. For example, assume that in a certain block, four sounds of "do-re-mi-fa" are played in this order in both the reference information 40 and the user performance information 50. In this situation, if the performance time of any one or more sounds is different between the reference information 40 and the user performance information 50 (for example, the difference in the performance time is a predetermined value or more), it can be said that a tempo deviation has occurred. Therefore, the evaluation unit 2040 detects a tempo deviation by comparing the reference information 40 and the user performance information 50 for each block and checking the above-mentioned situation.

[0176] <Output of evaluation information 30> The performance evaluation device 2000 outputs the evaluation information 30 in an arbitrary manner. For example, the performance evaluation device 2000 stores the evaluation information 30 in a storage device. As another example, the performance evaluation device 2000 displays the evaluation information 30 on a display device. As another example, the performance evaluation device 2000 may transmit the evaluation information 30 to another device. For example, when the performance evaluation device 2000 is realized as a server machine 80 that operates by receiving video data 20 from a user terminal 62 (see FIG. 12(b)), the performance evaluation device 2000 transmits the evaluation information 30 to the user terminal 62. Then, the evaluation information 30 can be viewed in the user terminal 62 in various ways.

[0177] Note that, instead of outputting the evaluation information 30 as is, the performance evaluation device 2000 may appropriately process and output the evaluation information 30. For example, the performance evaluation device 2000 generates an evaluation result screen using the evaluation information 30, and displays the evaluation result screen on a display device.

[0178] Fig. 24 is a first diagram illustrating an example of an evaluation result screen. The evaluation result screen 900 in Fig. 24 shows a graph generated from the evaluation information 30. In this example, a statistical score is calculated for each section and for each evaluation item. Also, in this example, the time transition of the statistical score for each evaluation item is represented by a line graph.

[0179] In addition, in Fig. 24, statistical scores that represent particularly poor evaluations are highlighted. Specifically, a threshold value for the statistical scores is indicated by a dashed line, and statistical scores that are smaller than the threshold value are highlighted by a dotted mark. Note that statistical scores that represent particularly good evaluations may be highlighted in a similar manner. Specifically, statistical scores that are larger than a predetermined threshold value are highlighted.

[0180] Fig. 25 is a second diagram illustrating an example of the evaluation result screen. The evaluation result screen 1300 in Fig. 25 shows a graph generated from the evaluation information 30. In this example, a statistical score is calculated for each finger used in playing and for each evaluation item.

[0181] By tallying up the item values ​​of each evaluation item for each finger used in playing in this way, it becomes easy to grasp, for example, the habits of each hand or finger. For example, looking at the graph in Figure 25, we can see that the scores for performances using the left hand are lower than the scores for performances using the right hand for both evaluation items A and B. This shows that the technique of playing with the left hand should be improved.

[0182] For example, the graph in Figure 25 shows that the scores for playing with the little finger are lower than the scores for playing with the other fingers for both the right and left hands. This shows that the technique of playing with the little finger should be improved.

[0183] The performance evaluation device 2000 may generate three-dimensional modeling images of hand shapes for the performance of the user 60 and the performance of the model performer, and output a screen including the images, or store the video data of the images in a storage device. The hand modeling images for the performance of the user 60 can be generated based on information such as the joint points of each finger and wrist obtained from the user performance information 50. Similarly, the hand modeling images for the model performer can be generated based on information such as the joint points of each finger and wrist obtained from the reference information 40.

[0184] The performance evaluation device 2000 preferably outputs evaluation information 30 together with the modeling image. Here, the evaluation result for each evaluation item is preferably displayed in association with a portion of the modeling image related to the evaluation item. For example, the evaluation result for the wrist height is displayed near the corresponding wrist. Similarly, the evaluation result for each finger joint point is displayed near the corresponding finger.

[0185] By showing the evaluation information 30 together with the hand modeling image in this way, the user 60 can intuitively visually grasp the difference between his or her own performance and that of the model performer. This allows the user 60 to easily recognize the points that need to be corrected (practiced).

[0186] In the above embodiment, the program can be stored and supplied to the computer using various types of non-transitory computer readable media. The non-transitory computer readable medium includes various types of tangible storage media. Examples of the non-transitory computer readable medium include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROM (Read Only Memory), CD-R, CD-R / W, and semiconductor memory (e.g., mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, and RAM (Random Access Memory). The program may also be supplied to the computer by various types of transitory computer readable media. Examples of the transitory computer readable medium include electric signals, optical signals, and electromagnetic waves. The transitory computer readable medium can supply the program to the computer via a wired communication path such as an electric wire and an optical fiber, or a wireless communication path.

[0187] Although the present invention has been described above with reference to the embodiment, the present invention is not limited to the above embodiment. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. [Explanation of symbols]

[0188] 10. Camera 20 Video Data 30 Evaluation Information 40 Standard Information 42 Performance time 44 item information 46 Evaluation Items 48 Item Values 50 User performance information 52 Performance time 54 item information 56 Evaluation Items 58 Item Values 60 users 62 User terminal 70 Target Instruments 72 Performance Data 80 Server Machine 90 Tempo setting screen 100 Tempo Information 110 fingers 120 keys 130 Reference Line 140 Hands 150 Reference Line 160 keys 170 straight line 180 straight line 190 keys 200 lines 210 Line Segments 300 Song selection screen 310 Search Area 320 Song selection area 330 Evaluation range designation area 400 Tables 402 Performance time 404 Evaluation Results 406 Evaluation Items 408 Rating Score 500 Tables 502 Performance time 504 Statistical Score 600 Tables 602 Evaluation items 604 Statistical Score 700 Tables 702 Subrange 704 Evaluation Results 706 Evaluation Items 708 Statistical Score 800 tables 802 Performance time 804 Evaluation Results 806 Item Group 808 Rating Score 900 Evaluation result screen 1000 Computers 1020 Bus 1040 Processor 1060 Memory 1080 Storage Device 1100 Input / Output Interface 1120 Network Interface 1300 Evaluation result screen 2000 Performance evaluation device 2020 Tempo Information Acquisition Department 2040 Evaluation Department

Claims

1. On the computer, A tempo information acquisition process for acquiring tempo information capable of identifying a designated tempo, which is a performance tempo desired by a user; an evaluation process for generating evaluation information including an evaluation of a performance by the user by comparing standard information, which indicates a standard for evaluating a performance when the performance is performed at the specified tempo, with user performance information, which indicates a performance of the musical piece by the user, the standard information is prepared for each piece of music in correspondence with a plurality of tempos, the standard information indicating a standard for evaluating a performance when the piece of music is performed at each tempo; The evaluation process is a program for obtaining standard information corresponding to the specified tempo from a plurality of standard information prepared for the piece of music to be evaluated, and generating the evaluation information using the obtained standard information.

2. the standard information is prepared for each piece of music in association with a specific tempo, the standard information indicating a standard for evaluating a performance when the piece of music is performed at that tempo; The evaluation process includes: acquiring information indicating the specific tempo and the reference information for the piece of music to be evaluated; converting the acquired reference information into reference information that represents an evaluation standard when the piece of music to be evaluated is played at the specified tempo based on a difference between the specific tempo and the specified tempo; The program according to claim 1 , further comprising: generating the evaluation information by utilizing the reference information obtained by the conversion.

3. a tempo information acquisition unit for acquiring tempo information capable of identifying a designated tempo, which is a performance tempo desired by a user; an evaluation unit that generates evaluation information including an evaluation of the performance by the user by comparing standard information that indicates an evaluation standard for a performance of a piece of music to be evaluated when the piece of music is played at the specified tempo with user performance information that indicates the performance of the piece of music by the user, the standard information is prepared for each piece of music in correspondence with a plurality of tempos, the standard information indicating a standard for evaluating a performance when the piece of music is performed at each tempo; The evaluation unit acquires standard information corresponding to the specified tempo from a plurality of standard information prepared for the piece of music to be evaluated, and generates the evaluation information using the acquired standard information.

4. A computer-implemented performance evaluation method, comprising the steps of: a tempo information acquisition step of acquiring tempo information capable of identifying a designated tempo, which is a performance tempo desired by a user; an evaluation step of generating evaluation information including an evaluation of the performance by the user by comparing, for a piece of music to be evaluated, standard information that indicates a standard for evaluating a performance when the piece of music is performed at the specified tempo with user performance information that indicates a performance of the piece of music by the user; the standard information is prepared for each piece of music in correspondence with a plurality of tempos, the standard information indicating a standard for evaluating a performance when the piece of music is performed at each tempo; The evaluation step is a performance evaluation method in which standard information corresponding to the specified tempo is obtained from a plurality of standard information prepared for the piece of music to be evaluated, and the evaluation information is generated using the obtained standard information.

Citation Information

Patent Citations

  • Musical amusement system

    JP1996305356A

  • Playing training apparatus and playing training method

    JP2002182553A

  • Device and method for tempo information output, computer program for tempo information output, device and method for touch information output, and computer program for touch information output

    JP2006065186A

  • Musical performance evaluation system and program of performance evaluation processing

    JP2006091633A

  • Scoring device and program

    JP2009104097A