Program, performance evaluation device, and performance evaluation method

Abstract

To provide new technology for evaluating performance of a musical instrument.SOLUTION: A camera 10 is installed to photograph a user 60 playing an object musical instrument 70. The user 60 plays an object musical piece with the object musical instrument 70. A performance evaluation device 2000 acquires video data 20 generated by the camera 10 photographing performance. The performance evaluation device 2000 generates user performance information 50 by analyzing the video data 20. The user performance information 50 indicates an item value of an evaluation item on performance of the object musical instrument 70 in performance by the user 60. The performance evaluation device 2000 acquires reference information 40 indicating a reference item value on the object musical piece. The performance evaluation device 2000 generates evaluation information 30 representing evaluation of performance by the user 60 by comparing the reference information 40 with the user performance information 50.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a technique for evaluating musical instrument performance. [Background technology]

[0002] Systems for evaluating musical instrument performance have been developed. A system that photographs the performance of an instrument from above and compares the photographed performance with a model performance to evaluate it. In the system of Patent Document 1, the ) Evaluation is made on the position, tilt, or width of the hand. Here, an evaluation is carried out to see if the fingering used in the performance is correct. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2006-091633 [Patent Document 2] Japanese Patent Application Laid-Open No. 2002-182553 Summary of the Invention [Problem to be solved by the invention]

[0004] In both Patent Documents 1 and 2, performance evaluation is performed using a video of the hands playing the instrument taken from above. Therefore, the information available for evaluating the performance is a top-down view of the performer's hands. The information is limited to what can be obtained by

[0005] The present invention has been made to solve such problems, and its object is to The goal is to provide new techniques for evaluating performances. [Means for solving the problem]

[0006] The program disclosed herein causes a computer to execute a reference information acquisition process that acquires reference information indicating reference item values ​​for evaluation items related to musical instrument performance in a piece of music played by a user; a video data acquisition process that acquires video data obtained by capturing an image of the user playing the piece of music on the musical instrument with a camera; an analysis process that generates user performance information indicating item values ​​for the evaluation items for the user's performance of the piece of music by analyzing an image of the user's hands included in the video data; and an evaluation process that generates evaluation information representing an evaluation of the user's performance by comparing the reference information with the user performance information.

[0007] The performance evaluation device disclosed herein includes a reference information acquisition unit that acquires reference information indicating reference item values ​​for evaluation items related to musical instrument performance in a piece of music played by a user; a video data acquisition unit that acquires video data obtained by capturing an image of the user playing the piece of music on the musical instrument with a camera; an analysis unit that generates user performance information indicating item values ​​for the evaluation items for the user's performance of the piece of music by analyzing images of the user's hands included in the video data; and an evaluation unit that generates evaluation information representing an evaluation of the user's performance by comparing the reference information with the user performance information.

[0008] A computer-implemented performance evaluation method according to the present disclosure includes a reference information acquisition step of acquiring reference information indicating reference item values ​​for evaluation items related to musical instrument performance in a musical piece played by a user, a video data acquisition step of acquiring video data by capturing an image of the user playing the musical piece on the musical instrument with a camera, an analysis step of generating user performance information indicating item values ​​for the evaluation items for the user's performance of the musical piece by analyzing an image of the user's hands included in the video data, and an evaluation step of generating evaluation information representing an evaluation of the user's performance by comparing the reference information with the user performance information. [Effects of the Invention]

[0009] According to the present invention, a new technique for evaluating musical instrument performance is provided. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating an example of an overview of a performance evaluation device according to a first embodiment. [Figure 2] 1 is a block diagram illustrating an example of the functional configuration of a performance evaluation device according to a first embodiment. [Figure 3] FIG. 1 is a block diagram illustrating an example of the hardware configuration of a computer that realizes a performance evaluation device. [Figure 4] FIG. 2 is a diagram illustrating an example of the position and orientation of a camera. [Figure 5] FIG. 1 is a diagram illustrating an example of how a performance evaluation device is used. [Figure 6] 1 is a flowchart illustrating the flow of processing executed by the performance evaluation device of the first embodiment. [Figure 7] FIG. 1 is a diagram illustrating an example of joint points and joint lines of a hand. [Figure 8] FIG. 10 is a diagram illustrating a method for calculating the height of a finger joint. [Figure 9] 10A and 10B are diagrams illustrating a method for calculating the magnitude of an angle between joint lines; [Figure 10] FIG. 10 is a diagram illustrating a method for calculating the height of a hand. [Figure 11] 10A and 10B are diagrams illustrating a method for calculating the height of the hand upper surface. [Figure 12] FIG. 10 is a diagram illustrating a method for calculating a wrist angle. [Figure 13] FIG. 10 is a diagram illustrating a method for calculating the depth of a keystroke. [Figure 14] FIG. 10 is a diagram illustrating an example of reference information in a table format. [Figure 15] FIG. 10 is a diagram illustrating an example of a designation screen for designating a target song. [Figure 16] FIG. 10 is a diagram illustrating an example of user performance information in a table format. [Figure 17]10 is a flowchart illustrating a process for calculating an evaluation score for each item of information. [Figure 18] FIG. 10 is a diagram illustrating an example of evaluation information in a table format. [Figure 19] FIG. 10 is a diagram illustrating an example of evaluation information showing statistical scores for each performance point. [Figure 20] FIG. 10 is a diagram illustrating an example of evaluation information indicating a statistical score for each evaluation item. [Figure 21] FIG. 10 is a diagram illustrating an example of evaluation information showing statistical scores for each subrange and for each evaluation item. [Figure 22] FIG. 10 is a diagram illustrating an example of statistical values ​​of evaluation scores tallied for each group. [Figure 23] FIG. 10 is a diagram illustrating an example of an evaluation result screen in the first embodiment. [Figure 24] FIG. 10 is a block diagram illustrating the functional configuration of a performance evaluation device according to a second embodiment. [Figure 25] 10 is a flowchart illustrating the flow of processing executed by the performance evaluation device of the second embodiment. [Figure 26] FIG. 10 is a diagram illustrating an example of reference information in a table format according to the second embodiment. [Figure 27] FIG. 10 is a diagram illustrating an example of evaluation information showing statistical scores for sound intensity and sound duration, calculated for each finger used in playing. [Figure 28] FIG. 10 is a diagram illustrating an example of an evaluation result screen in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings. The same components are denoted by the same reference numerals, and duplicated explanations will be omitted.

[0012] [Embodiment 1] <Summary> FIG. 1 is a diagram illustrating an example of an outline of a performance evaluation device 2000 according to the first embodiment. To facilitate understanding of the evaluation device 2000, an example of the operation of the performance evaluation device 2000 will be conceptually explained below. 1, and the operation of the performance evaluation device 2000 is not limited to that shown in FIG. .

[0013] The performance evaluation device 2000 generates evaluation information 30 representing an evaluation of a musical instrument performance performed by a user 60. Here, the instrument played by the user 60 is called the target instrument 70. 0 is a keyboard instrument such as a piano, organ, or synthesizer.

[0014] The evaluation information 30 is generated by capturing the musical instrument performance of the user 60 with the camera 10. This is done by analyzing the video data 20. The camera 10 captures the performance of the user 60. For example, the camera 10 is set to capture the side of the hand of the user 60 from diagonally above. It is installed in a position that looks down from above.

[0015] The performance evaluation device 2000 compares the actual performance performed by the user 60 with a predetermined standard. The comparison generates evaluation information 30. The comparison is performed for one or more evaluation items. The evaluation items can be understood by observing the performance of the user 60 from the side. Such evaluation items include, for example, knuckle height. Examples include:

[0016] More specifically, for example, the performance evaluation device 2000 operates as follows. The device 2000 acquires the video data 20 generated by the camera 10 and By analyzing each video frame 22 that constitutes the music data 20, user performance information 50 is generated. The user performance information 50 is information indicating the item values ​​of each evaluation item obtained from the user's performance. For example, suppose that "fingernail height" is defined as an evaluation item. The performance evaluation device 2000 detects the hand image of the user 60 from the video frame 22. The height of each finger joint of the user 60 is calculated using the above formula.

[0017] Furthermore, the performance evaluation device 2000 evaluates the musical piece played by the user 60 on the target instrument 70 by: The standard information 40 is acquired. The standard information 40 includes the standard information for each evaluation item regarding a specific song. The following shows the item values ​​that are the standard (for example, ideal item values). The songs are called target songs. Target songs can also be expressed as "songs that you want to use as the subject of evaluation."

[0018] The performance evaluation device 2000 compares the reference information 40 with the user performance information 50. As will be described later, the performance evaluation device 2000 generates the evaluation information 30. The item value indicated by the reference information 40 and the item value indicated by the user performance information 50 for one evaluation item are By comparing the reference information 40 and the user performance information 50, the degree of match between the reference information 40 and the user performance information 50 is determined. Evaluation information 30 indicating the degree of match is generated.

[0019] <Example of effects> According to the performance evaluation device 2000 of this embodiment, the performance of the user 60 is photographed from the side. Video data 20 is acquired from a camera 10 arranged to The performance of the user 60 is evaluated by analyzing the above. For evaluation items whose item values ​​can be grasped by looking at them from the side, the values ​​obtained from video data 20 The item values ​​are calculated, and the performance of the user 60 is evaluated using the item values.

[0020] Therefore, in the performance evaluation device 2000, the performance of the user 60 is evaluated based on the user's 60's It utilizes the characteristics of the performance that can be grasped by looking at the performance from the side. In the first and second inventions, the performance of the user is evaluated by looking at the top of the user's hand. The performance characteristics that can be grasped by observing the performance from the side are not reflected in the evaluation of the performance. Therefore, according to the performance evaluation device 2000, the following points were not taken into consideration in the inventions of the cited documents 1 and 2: The performance of the user 60 can be evaluated taking into consideration the characteristics of the performance of the user 60.

[0021] The performance evaluation device 2000 of this embodiment will be described in more detail below.

[0022] <Example of functional configuration> FIG. 2 is a block diagram illustrating the functional configuration of the performance evaluation device 2000 according to the first embodiment. In this example, the performance evaluation device 2000 includes a reference information acquisition unit 2020, a video data acquisition unit 2030, and a The reference information acquisition unit 2020 includes a unit 2040, an analysis unit 2060, and an evaluation unit 2080. The video data acquisition unit 2040 acquires the reference information 40 for the target song. The analysis unit 2060 acquires the hand or hand motion of the user 60 included in the video data 20. By analyzing the image of the musical instrument, the user performance information 50 is generated. The information 40 is compared with the user performance information 50 to generate evaluation information 30.

[0023] <Example of hardware configuration> Each functional component of the performance evaluation device 2000 is implemented by hardware (e.g., It may be realized by a combination of hardware and software (e.g., hardwired electronic circuits) or It is implemented in combination with air (e.g., a combination of an electronic circuit and a program that controls it). Hereinafter, each functional component of the performance evaluation device 2000 may be implemented as hardware and software. The case where it is realized in combination with air will be further explained.

[0024] FIG. 3 shows the hardware configuration of the computer 1000 that implements the performance evaluation device 2000. 1 is a block diagram illustrating an example of a computer 1000. For example, the computer 1000 may be a desktop computer such as a PC (Personal Computer) or a server machine. In addition, the computer 1000 may be a smart Portable computers such as phones and tablets. The computer 1000 may be a camera 10. In addition, for example, If the instrument being played is an electronic instrument, the computer 1000 The computer may be a computer that is

[0025] The computer 1000 is a dedicated computer designed to realize the performance evaluation device 2000. It may be a computer or a general-purpose computer. By installing a specific application on the computer 1000, The application 1000 realizes each function of the performance evaluation device 2000. It is composed of a program for realizing the functional components of the performance evaluation device 2000.

[0026] The computer 1000 includes a bus 1020, a processor 1040, a memory 1060, a storage storage device 1080, input / output interface 1100, and network interface The bus 1020 connects the processor 1040, the memory 1060, the storage The image device 1080, the input / output interface 1100, and the network interface The processor 1120 is a data transmission path for transmitting and receiving data to and from each other. The method of connecting the sensors 1040 and the like to each other is not limited to bus connection.

[0027] The processor 1040 includes a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), various processors such as FPGAs (Field-Programmable Gate Arrays) The memory 1060 is a main storage device realized using RAM (Random Access Memory) or the like. The storage device 1080 may be a hard disk, a solid state drive (SSD), It is an auxiliary storage device realized using a memory card or ROM (Read Only Memory), etc. do.

[0028] The input / output interface 1100 connects the computer 1000 to an input / output device. The network interface 1120 is an interface for connecting to a computer. This is the interface for connecting the 1000 to the network. It may be a LAN (Local Area Network) or a WAN (Wide Area Network). stomach.

[0029] The storage device 1080 stores programs that realize the various functional components of the performance evaluation device 2000. The processor stores the programs (programs that implement the aforementioned applications). 1040 reads this program into memory 1060 and executes it to perform performance evaluation. Each functional component of the device 2000 is realized.

[0030] The performance evaluation device 2000 may be realized by one computer 1000 or by multiple computers. In the latter case, each computer 100 The configuration of the 0's does not have to be the same, but can be different.

[0031] <About Camera 10> The camera 10 can capture the performance of the user 60 from the side and generate video data. The camera 10 is a dedicated camera for capturing images of a user playing a musical instrument. It may be a camera, or a general-purpose camera (for example, a camera installed in a smartphone, etc.). etc.)

[0032] If the computer 1000 is implemented as a computer other than the camera 10, for example, The controller 10 receives data via an input / output interface 1100 or a network interface 1120. The computer 1000 is connected to the video device. It is sufficient that the camera 10 can acquire the data 20, and it is not necessary for the camera 10 to be connected so that it can communicate with the camera 10. The method by which the evaluation device 2000 acquires the video data 20 will be described later.

[0033] When the performance of the user 60 is evaluated for each of the left and right hands, for example, The camera 10 is placed in a position where the hand is closest to the user, and the camera 10 is placed in a position where the right hand of the user 60 is closest to the user. It is preferable to provide both the camera 10 and the camera 10. 4(a) is a plan view of the target musical instrument 70, and FIG. 4 is a front view of the target musical instrument 70. In FIG. 4, the longitudinal direction of the keyboard of the target musical instrument 70 is the x-axis direction. The lateral direction of the keyboard of the target musical instrument 70 is the y-axis direction, and the vertical direction is the z-axis direction.

[0034] The camera 10-1 captures at least the side of the fingers of the left hand of the user 60 from the left side (leftward of the right hand). In FIG. 4, the camera 10- 1 is placed so as to look down on the target instrument 70 in the +x direction.

[0035] On the other hand, camera-2 captures at least the side of the fingers of the right hand of the user 60 from the right side (more clearly than the left hand). In Figure 4, camera 1 is installed so that it can take pictures from the direction where the right hand side comes forward. 0-2 is placed so as to look down on the target instrument 70 in the -x direction.

[0036] The shooting direction of the camera 10 is not limited to the example shown in Fig. 4. For example, The horizontal component of the shooting direction of the camera 10-2 does not need to be perfectly parallel to the x-axis, but may be parallel to the y-axis. It may contain ingredients.

[0037] The specific installation position and installation posture of the camera 10 may be determined arbitrarily by the user. In the former case, the camera 10 captures the user's The camera is set up in a position and posture that allows the performance of the camera 60 to be photographed from the side.

[0038] On the other hand, if the installation position and installation posture of the camera 10 are predetermined, for example, and a device that can be installed at a predetermined position and that can install the camera 10 in a predetermined posture. This equipment is used in the target instrument when the target instrument is provided (for sale, lease, etc.). It may be installed in the instrument itself, or it may be installed after the instrument is provided (for example, by purchasing a separately sold adapter). provided as an attachment and installed by the user).

[0039] For example, while viewing the video data generated by camera 10, By adjusting the installation position and installation posture of the camera 10, the camera 10 is installed in a predetermined position and posture. For example, a predetermined reference mark may be provided at a predetermined position on the target instrument. The camera 10 captures the reference mark on the musical instrument, and the video obtained from the camera 10 While looking at the data, the camera 10 photographs the reference mark in a predetermined position, size, and shape. The position and attitude of the camera 10 are adjusted so that the image can be captured.

[0040] The number of cameras 10 used to evaluate the performance is not limited to two. When evaluating the performance of only one hand, a camera 10 capable of photographing the hand to be evaluated is placed in the Alternatively, for example, the same hand may be photographed at different angles of depression. Alternatively, multiple cameras 10 may be provided for each of the right and left hands.

[0041] <Example of Usage of Performance Evaluation Device 2000> To facilitate understanding of the performance evaluation device 2000, an example of how the performance evaluation device 2000 is used will be given below. The usage modes described here are merely examples, and the specific usage modes of the performance evaluation device 2000 are as follows: The usage is not limited to the example described here.

[0042] FIG. 5 is a diagram illustrating an example of how the performance evaluation device 2000 is used. In FIG. 5(a), The performance evaluation device 2000 is a mobile terminal (user terminal 62) operated by a user 60. For example, in order to realize each functional component of the performance evaluation device 2000 described above, By installing this program on the user terminal 62, the performance evaluation can be performed on the user terminal 62. It is conceivable to realize a valuation device 2000.

[0043] The user terminal 62 is a camera that captures the user 60 playing the target musical instrument 70 from the left. The camera 10 captures the user 60 playing the target instrument 70 from the right. The user terminal 62 then acquires the video data 20 from each of the The video data 20 is analyzed to generate user performance information 50, and the user performance information 50 The evaluation information 30 is generated by comparing the evaluation information 30 with the reference information 40.

[0044] On the other hand, in FIG. 5(b), the performance evaluation device 2000 receives a request from the user terminal 62. This is realized by a server machine 80 that accepts and operates the user. The user terminal 62 is used as an interface for using the performance evaluation device 2000. Specifically, the user terminal 62 acquires video data 20 from two cameras 10. Then, the video data 20 is transmitted from the user terminal 62 to the server machine 80. For example, the video data 20 is attached to a request for evaluation of the performance of the user 60. The server machine 80 analyzes the acquired video data 20 and generates user performance information 50. The user's performance information 50 is then compared with the reference information 40 to generate evaluation information 30. Then, the server machine 80 sends a response including the evaluation information 30 to the user terminal 62. Believe.

[0045] <Processing flow> FIG. 6 illustrates the flow of processing executed by the performance evaluation device 2000 of the first embodiment. 1 is a flowchart. The reference information acquisition unit 2020 acquires the reference information 40 (S102). The video data acquisition unit 2040 acquires the video data 20 (S104). The evaluation section 60 generates the user performance information 50 using the video data 20 (S106). The unit 2080 generates evaluation information 30 using the reference information 40 and the user performance information 50 ( S108).

[0046] The processing flow executed by the performance evaluation device 2000 is not limited to that shown in FIG. For example, before S102, S104 or both S104 and S106 are executed. Furthermore, S104 and S106 may be executed in parallel with S102.

[0047] <Evaluation items> The performance by the user 60 is evaluated based on a number of evaluation criteria. Any item that reflects good or bad can be adopted. However, the evaluation items At least one of them uses an image of the performance of the user 60 taken from the side to calculate the item value. In other words, the evaluation items include one. These include at least one item whose value is difficult to calculate.

[0048] The item values ​​of various evaluation items described below are calculated using, for example, the joint points and joint lines of the hand. A joint point is a point that indicates the position of a joint. A joint line is a line that connects adjacent joint points. It is a straight line. Figure 7 is a diagram showing an example of the joint points and joint lines of the hand. In Figure 7, the joint points are It is represented by a circle.

[0049] Below, the item values ​​can be calculated by using an image of the performance of user 60 taken from the side. Specific examples of the evaluation items will be explained below.

[0050] <<Knuckle height>> "Finger joint height" is an evaluation item that indicates the height position of the finger joints. The joint points used by 2000 for evaluation may be all joint points or only some of them. Good too.

[0051] For example, the height of a finger joint is determined by the positional relationship between the finger joint point and a reference plane. For example, the top surface of the keyboard when not being pressed is determined by the base surface. In addition, for example, when the key is pressed down to the lowest position, The upper surface may be treated as the reference surface. There are various specific methods for calculating the height of the finger joints. methods can be used.

[0052] Preferably, the height of the finger joint is calculated for the finger in a typing state, provided that: The height of the finger joint may also be calculated for fingers that are not typing.

[0053] FIG. 8 is a diagram illustrating a method for calculating the height of a finger joint. For ease of explanation, in FIG. In the video frame 22, the finger 110 and the area around the key 120 being pressed by the finger 110 are Only the image areas of the edges are shown.

[0054] In Figure 8, a reference line 130 is used to represent the position of the reference plane. is the key next to the key 120 on which the finger 110 is placed that is farther from the camera 10. It is a straight line that overlaps with the long side of the key. It is preferable to use the long side of the top surface of the key 120. The method of expressing the position of the reference plane is as follows. The method is not limited to the method using the reference line 130. For example, the key adjacent to the key 120 may be The long side of the key that is closer to the camera 10 may be used as the reference line that indicates the position of the reference plane. .

[0055] Figure 8(a) and (b) show different methods for calculating the height of the finger joints. In Figure 8(a), the height of joint point X is represented by the length of line segment XA. Point A is The intersection of a line extending vertically downward from the articulation point X on the video frame 22 with the reference line 130 On the other hand, in Figure 8(b), the height of joint point X is expressed by the length of line segment XB. Point B is the foot of a perpendicular line drawn from joint point X to reference line 130.

[0056] In the example of FIG. 8, the camera 10 is installed so as to capture an image of the side of the hand from diagonally above. In this case, the vertical direction of the video frame 22 and the downward movement of the reference line 130 in FIG. The direction of the perpendicular line drawn is different from the vertical direction in the real world. As shown above, the item value of the evaluation item "fingertip height" is the height in the vertical direction in the real world. For directions that have a vertical component (i.e., directions that are not horizontal), It is sufficient if it represents the distance from the joint point to the reference plane. The same applies to "wrist height."

[0057] However, the performance evaluation device 2000 uses the height of the finger joints in the vertical direction in the real world. In this case, for example, the performance evaluation device 2000 may handle the length of the video frame 22. From the finger joint positions and reference plane in the 2D coordinate system (hereafter referred to as the image coordinate system), the 3D By calculating the position of the finger joints and the reference plane in the coordinate system (hereafter referred to as the world coordinate system), Calculate the height of the finger joint in the hand coordinate system.

[0058] Transformation from 2D coordinates to 3D coordinates is used, for example, in stereo vision and depth This can be achieved by using a camera. In the former case, the performance of user 60 is photographed from the left. Two cameras 10 are installed, one for taking pictures from the right side and one for taking pictures from the left side. To calculate the height of the knuckles on the right hand of the user 60, two cameras 110 and 112 are used to capture the user 60 from the right. Using the video frames 22 obtained from each of these, the height of the finger joints in the world coordinate system is calculated. When using a depth camera, the depth is calculated for each pixel on the video frame 22. Therefore, the distance to the object being photographed can be determined. From the 2D coordinates of the finger joints, the 3D coordinates of the finger joints in the world coordinate system can be determined. can.

[0059] In order to define the world coordinate system, a predetermined mark ( It is preferable to provide a mark (size and shape) on the target instrument at a predetermined position. By taking a picture of a mark with a known size and shape with a camera, the image obtained from the camera The technology to determine the relationship between the 2D coordinate system and the world coordinate system can be based on existing technology. can be done.

[0060] <<The size of the angle between the joint lines>> The "size of the angle between joint lines" is the size of the angle formed by two joint lines with a common endpoint. This is an evaluation item that indicates the size. Figure 9 is a diagram illustrating a method for calculating the size of the angle between the joint lines. For ease of explanation, FIG. 9 shows only a portion of the video frame 22 surrounding the finger 110. Only the image area is shown. The size of the angle between the joint lines is the size of the finger when typing. However, it is preferable to calculate the joints of the fingers when they are not typing. The magnitude of the angle between the lines may be calculated.

[0061] In Figure 9, joint points C and D are adjacent joint points. The line connecting them is the joint line CD. Similarly, joint points D and E are also adjacent joint points. The straight line connecting these joint points is the joint line DE. The angle between the line ED is α. For example, let us consider the angle between the joint line CD and the joint line D. It can be used as an item value for the evaluation item of the size of the angle formed by E.

[0062] For example, the value of the item "the size of the angle between two joint lines" can be The magnitude of the angle formed by these two joint lines in the world coordinate system is used. In this case, the performance evaluation device 2000 may calculate the coordinates of each finger joint on the video frame 22 as By converting the coordinates into the world coordinate system, two joint lines in the world coordinate system are identified. This allows us to determine the size of the angle between the two joint lines in the world coordinate system. It is possible.

[0063] <<Wrist height>> "Wrist height" is an evaluation item that indicates the position of the wrist in the height direction. For example, the height of the wrist is The positional relationship between the wrist joint and the reference plane is determined. The same reference plane can be used. The wrist height should be adjusted so that at least one finger is used to type. It is preferable that the calculation be performed for the hand in the state where the key is pressed, but the keystrokes are not performed with any fingers. The wrist height may also be calculated for an unwound hand.

[0064] FIG. 10 is a diagram illustrating a method for calculating the height of the wrist. 10, in the video frame 22, the hand 140 and the area around the key that the hand 140 is typing are Only the image area of ​​is shown.

[0065] In FIG. 10, the joint point of the wrist of the hand 140 is Y. Also, in FIG. 10, the position of the reference plane A reference line 150 is used to indicate the position of the key being pressed. The line overlaps with the long side of the key next to 60 that is farther from the camera 10. As in the case of calculating the joint height, the baseline is the state when no key is being pressed. It is also possible to use the long side of the top surface of the key. It is not limited to the key struck by the finger closest to the mela 10 or the adjacent key.

[0066] Figure 10(a) and (b) show different methods for calculating the wrist height. In FIG. 10(a), the height of the wrist is determined by the distance of the line segment YG. Point G is A line drawn vertically downward from the wrist joint point Y in video frame 22 and the reference line 150 On the other hand, in Figure 10(b), the height of the wrist is determined by the distance of the line segment YH. Point H is the foot of a perpendicular line drawn from wrist joint point Y to reference line 150.

[0067] The performance evaluation device 2000 also evaluates the height of the wrist in the same way as the height of the finger joints. The performance evaluation device 2000 may also handle the height in the vertical direction of the image. From the wrist position and reference plane in the coordinate system, the wrist position and reference plane in the world coordinate system are calculated. Then, the distance from the wrist position in the world coordinate system to the reference plane is It is treated as a height.

[0068] <<Hand height>> "Height of the upper surface of the hand" is an evaluation item that indicates the height position of the upper surface of the hand. The height of the finger is determined by the positional relationship between the third joint of the middle finger (see Figure 7) and the reference plane. The same reference plane as that for determining the height of the finger joints can be used. It is preferable that the calculation be performed for a hand in which at least one finger is typing. The height of the hand surface is also calculated for hands where no typing is being performed with any fingers. That's fine.

[0069] FIG. 11 is a diagram illustrating a method for calculating the height of the hand upper surface. The hands 140, keys, and reference lines 150 are the same as those shown in FIG. In addition, in FIG. 11, the position of the upper surface of the hand 140 is the third joint point of the middle finger of the hand 140. It is represented by W.

[0070] Figure 11(a) and (b) show different methods for calculating the height of the hand. In Figure 11(a), the height of the hand is determined by the distance of the line segment WK. is the distance between a line drawn vertically downward from the hand's upper surface W to the video frame 22 and the reference line 150. On the other hand, in Fig. 11(b), the height of the hand is determined by the distance of the line segment WL. Point L is the foot of a perpendicular line drawn from the top surface of the hand W to the reference line 150.

[0071] The performance evaluation device 2000 also measures the height of the upper surface of the hand in the same way as the height of the finger joints and wrist. In addition, the performance evaluation device 2000 may handle the height in the vertical direction in the real world. is the position of the hand's top surface in the world coordinate system, calculated from the position of the hand's top surface in the image coordinate system and the reference plane. Then, the distance from the hand's top surface in the world coordinate system to the reference plane is calculated. The distance is treated as the height of the top of the hand.

[0072] <<Wrist angle>> "Wrist angle" is an evaluation item that indicates the degree of bending of the wrist. For example, the wrist angle is It can be determined by the size of the angle formed by the line representing the posture of the arm and the line representing the posture of the top of the hand. The wrist angle is calculated for a hand in which at least one finger is typing. However, for hands where no typing is being performed with any fingers, Alternatively, the wrist angle may be calculated.

[0073] FIG. 12 is a diagram illustrating a method for calculating the wrist angle. 12, only the image area of ​​the video frame 22 surrounding the hand 140 is shown. The angle of the wrist is expressed by the angle β formed by the lines 170 and 180. is a line that represents the posture of the arm. On the other hand, line 180 is a line that represents the posture of the upper surface of the hand. .

[0074] For example, line 170 is a line that detects an edge representing the upper surface of an arm from video frame 22 and Line fitting can be performed by connecting any two points on the edge or by using three or more points on the edge. Similarly, the line 180 can be obtained by 2, the edge representing the top surface of the hand is detected, and any two points on the edge are connected, or the edge is By using three or more points on the map and performing a straight line fitting, Cut.

[0075] The angle of the wrist may also be expressed in terms of the angle in the world coordinate system. In this case, for example, the performance evaluation device 2000 calculates a straight line representing the arm posture in the world coordinate system. Calculate the lines that represent the posture of the top surface of the hand, and calculate the angle between these lines as the wrist angle. In addition, in the world coordinate system, the posture of the arm and the posture of the top surface of the hand are not linear but planar. In this case, the angle between these two planes can be treated as the wrist angle. This can be done.

[0076] <<Keystroke depth>> "Keystroke depth" is an evaluation item that indicates the degree to which the keys are pressed when playing. The length can be expressed by the positional relationship between the key in normal operation and the key when it is pressed.

[0077] FIG. 13 is a diagram illustrating a method for calculating the depth of keystroke. , only the image area of ​​video frame 22 surrounding key 190 is shown.

[0078] Key 190 is the key that is being pressed. Line segment 200 is the key that is being pressed normally (not pressed). The key 190 in the locked state is positioned at the top of the key 190, and the key 190 is positioned at the top of the key 190. On the other hand, the line segment 210 is the long side of the top surface of the key 190 when it is being pressed. That is, it represents the long side farther from the camera 10. Note that the line segment 200 is not keyed. Alternatively, the key 190 may be located on the long side of the top surface of the key next to the key 190 that is farther from the camera 10 .

[0079] Figures 13(a) and (b) show examples of different calculation methods for keystroke depth. In FIG. 13(a), the depth of the keystroke is represented by the size of the angle formed by the line segment 200 and the line segment 210. On the other hand, in Figure 13(b), the depth of the keystroke is expressed by the length of the line IJ. Point I is the end point of the line segment 200 that is closer to the body of the user 60. J is the end point of the line segment 210 that is closer to the body of the user 60 .

[0080] Note that the depth of the keystroke may also be expressed using a value in the world coordinate system. The evaluation device 2000 converts the line segments 200 and 210 in the image coordinate system into the world coordinate system. The size of the angle between these line segments and the distance between their endpoints are calculated based on the keystroke depth. The points to be used are not limited to the endpoints, but are used to find the line segment based on the similarity relationship. Any point on line segment 200 and line segment 210 may be used.

[0081] <Acquisition of standard information 40: S102> The standard information 40 is information indicating standard item values ​​for each evaluation item regarding a predetermined song. The standard item value is, for example, obtained by analyzing the performance of an advanced person such as a teacher. The item values ​​obtained are as follows. The standard information 40 is generated in advance for each piece of music that can be evaluated, and The data is stored in a storage device accessible from the device 2000.

[0082] Here, the item values ​​of the various evaluation items described above change as the performance progresses. The reference information 40 is time-series data that indicates the item values ​​of each evaluation item in time series. The information 40 is a combination of "the performance time and the item value of each evaluation item corresponding to that performance time." The performance time is information indicating the start time of the music piece (time 0 The start of a song is expressed as a relative point in time, for example, the first This is the point at which the performance (keystroke) takes place.

[0083] 14 is a diagram showing an example of the reference information 40 in a table format. Each point 42 has item information 44. The item information 44 is used to record the performance at the corresponding performance point. For each evaluation item, a pair of evaluation item 46 and item value 48 is shown. The item value 48 indicates the identification information for identifying the evaluation item (for example, the name of the evaluation item). Indicates the item value of the value item.

[0084] The reference information acquisition unit 2020 identifies a target song and acquires reference information 40 about the song. There are various methods for the reference information acquisition unit 2020 to identify target songs. The information acquisition unit 2020 receives a user input specifying a target song. The user who makes the setting may be any person, and does not have to be the user 60 who is the performer.

[0085] 15 is a diagram showing an example of a selection screen for selecting a target song. The song selection screen 300 is It includes a search area 310, a song selection area 320, and an evaluation range specification area 330. Area 310 is used to search for target songs using search words such as song titles and attributes (genres, etc.). This is an input interface for searching.

[0086] The search for the target song is not limited to the search using search words. The acquisition unit 2020 acquires the audio data of the target song and searches the song database using the audio data. The target song may be identified by searching for the song.

[0087] The search results are displayed in the song selection area 320. The user can select a song in the song selection area 320. In the example in Figure 15, Before the search, the song "Song F" is selected. The area 320 may or may not display candidate songs. In this case, for example, the song selection area 320 displays all songs that can be selected as target songs. In addition, for example, the song selection area 320 may also display songs that have been selected as target songs. A list of songs that have been played (that is, a history of songs played up to now) may be displayed.

[0088] The evaluation range designation area 330 is used when you want to evaluate only a part of a song. In this example, both the start and end positions can be specified. The portion of songs selected is called the "evaluation range."

[0089] There are various specific methods for specifying the evaluation range. For example, a song can be divided into multiple sections in advance. (such as intro and chorus) and then user input determines the evaluation range for one or more sections. In this way, the evaluation range can be specified by section. In this case, the standard information 40 can specify which section each item information 44 is included in. For example, the reference information 40 includes information about all sections included in the corresponding song. For each section, you can see the name of the section, the start of the section, The evaluation scope is specified by section. In this case, the reference information acquisition unit 2020 acquires the item information 44 included in the reference information 40 of the target song. That is, from the start of the first section of the assessment range to the end of the last section of the assessment range. Each item information 44 included in the range of points is used for evaluation.

[0090] The method of specifying the evaluation range is not limited to the method of specifying a section. The range may be specified in units of measures. When specifying the evaluation range in units of measures like this, The reference information 40 includes information that can identify which measure each item information 44 is included in. For example, the reference information 40 may include the following information for each measure included in the corresponding piece of music: The score also includes the information "number, the start time of the measure, and the end time of the measure." When the value range is specified in units of bars, the reference information acquisition unit 2020 acquires the reference information 4 of the target song. Of the item information 44 included in 0, from the start of the first measure of the evaluation range to the end of the evaluation range Each item information 44 included in the range at the end of the subsequent measure is used for evaluation.

[0091] However, in the case of music in which the length of a measure is constant, if the length of one measure is known, the measure number can be calculated. Therefore, when dealing with music with a fixed bar length, Alternatively, the reference information 40 may include the length of one measure, rather than the time range of each measure. In this case, the reference information acquisition unit 2020 uses the length of one measure indicated by the reference information 40 to determine the evaluation criteria. The start time of the first measure in the range and the end time of the last measure in the range are calculated. The quasi-information acquisition unit 2020 determines whether the performance time 42 is included in the range from the start time to the end time. The item information 44 that falls within the evaluation range is identified as the item information 44 that falls within the evaluation range and is used for evaluation.

[0092] The music selection screen 300 further includes an input interface that allows various conditions relating to evaluation to be specified. For example, the evaluation criteria may include the age group of the user. The age group of the user 60 is, for example, either "adult" or "child." It may be possible to specify more precisely, such as "infant" or "elementary school student." If the age group of the user 60 can be specified, the reference information 40 can be prepared for each age group. This is because the size of the hands of players varies depending on their age group, and this difference affects the evaluation value. To have an impact.

[0093] Another example of an evaluation condition is the type of instrument. "stick piano," "electronic piano," "organ," or "synthesizer" In this way, when the type of musical instrument can be specified, the reference information 40 is used for each type of musical instrument. This is because the hardness of the keys (the force required to strike the keys) varies depending on the type of instrument. This is because they differ and this difference affects the evaluation value.

[0094] The input interface for specifying the evaluation conditions is separate from the music selection screen 300. The screen may be provided as:

[0095] The music selection screen 300 is displayed on a display device that can be viewed by the user. For example, this display device may be an application that implements the functions of the performance evaluation device 2000. A computer (user terminal 62, camera 10, or target musical instrument 7) on which the The information is displayed on the touch panel display installed in the car (such as the car's internal clock).

[0096] In addition, the reference information acquisition unit 2020 does not specify the target songs, but acquires the reference information for all songs. In this case, for example, the performance evaluation device 2000 may acquire the criterion information 40 for each piece of music. The performance of the user 60 is evaluated using each piece of information 40. In this case, the highest evaluation is The music piece that has been played is considered to be the music piece that has been played by the user 60. The performance evaluation device 2000 then generates evaluation information 30 for the piece of music that has received the highest evaluation. Only this is used as the final output.

[0097] <<How to generate reference information 40>> For example, the reference information 40 may be used to analyze the video data 20 in which a performance by a model performer is recorded. Here, the device that generates the reference information 40 is called a reference information generator. The reference information generating device is implemented by the same computer as the performance evaluation device 2000. may be implemented on different computers.

[0098] The reference information generating device includes video data 20 recording the performance of the model performer. and extracting the video data 20 from the video frame 22 where the key was pressed. Further, the reference information generating device extracts the extracted video frames 22. The frame 22 is analyzed to calculate the item value for each evaluation item. For each extracted video frame 22, the generating device The performance time and the item information ("evaluation item, item value") obtained from the video frame 22 By associating the video frame with the reference information 40, the reference information 40 is generated. We will also explore how to identify the performance points corresponding to the music and calculate the item values ​​for each evaluation item from the video frames. The method for doing this will be described later in the description of the method for generating the user performance information 50.

[0099] The reference information 40 is generated based on information obtained from the performances of a plurality of model performers. For example, the reference information generating device may generate a plurality of model performers in the above-described manner. After generating the reference information 40 for each, statistical processing is performed on the generated reference information 40. By doing so, the reference information 40 to be used for evaluating the performance is generated. It is possible to use statistical values ​​(such as average values) of the item values ​​obtained for each.

[0100] <Acquisition of video data 20: S104> The video data acquisition unit 2040 acquires the video data 20 (S104). There are various methods available for obtaining the video data generated by the camera. (When the performance evaluation device 2000 is realized by a computer other than the camera 10), video data The acquisition unit 2040 receives the video data 20 transmitted from the camera 10 and accessing a storage device in which the video data 20 is stored and reading the video data 20; In this way, the video data 20 is acquired. The storage device may be provided either inside or outside the camera 10. When the performance evaluation device 2000 is realized by the camera 10, the video data acquisition unit 2040 00 (for example, storage device 1080) By reading, the video data 20 is obtained.

[0101] <Generation of user performance information 50: S106> The analysis unit 2060 analyzes the video data 20 to generate the user performance information 50. The user performance information 50 includes information about a performance performed by a user 60, corresponding to the time of the performance. The evaluation items and item values ​​at the time of the performance are displayed.

[0102] FIG. 16 is a diagram showing an example of the user performance information 50 in a table format. The structure of the performance information 50 is the same as the structure of the reference information 40 shown in FIG. The user performance information 50 includes item information 54 for each performance time point 52. The item information 54 includes: For the performance at the corresponding performance time, a pair of evaluation item 56 and item value 58 is The evaluation item 56 is an identification information for identifying the evaluation item (for example, the name of the evaluation item). The item value 58 indicates the item value of the corresponding evaluation item.

[0103] The analysis unit 2060 analyzes each of the plurality of video frames 22 included in the video data 20. The item value of each evaluation item is calculated for each video frame. The video frames 22 may be all the video frames 22 included in the video data 20, or a selected set of video frames 22. In the latter case, for example, the analysis unit 2060 may Video frame 22 for every frame (e.g., one video frame 22 for every three frames) ), and the item values ​​are calculated. When the reference information 40 is acquired, the analysis unit 2060 analyzes the reference information 40 at each performance time point 42 Alternatively, only the video frame 22 at the time indicated by may be used as the calculation target for the item value. For example, the analysis unit 2060 may analyze a predetermined period including the time indicated by the performance time 42 (the time The video frames 22 corresponding to the event (predetermined time before and after the event) may be used as the calculation targets for the item values.

[0104] In order to calculate the item value of each evaluation item from the video frame 22, the analysis unit 2060 The image analysis is performed on the video frame 22. Specifically, the analysis unit 2060 analyzes the video frame 22 based on the evaluation items. The information required to calculate the item value of the video frame 22 is extracted from the video frame 22. 2060 calculates the item value of each evaluation item using the extracted information. 060 indicates, for each video frame 22, the performance time corresponding to that video frame 22. and a combination of the item values ​​calculated for each evaluation item using the video frame 22. The user performance information 50 indicating the above is generated.

[0105] The method for calculating the item value of each evaluation item from the video frame 22 uses the data shown in FIGS. 8 to 13. The information required to calculate the item value can be, for example, the fingertip position of each finger. The position of the joints, the position of the wrist of each hand, the various reference lines mentioned above, the straight line representing the posture of the arm or the top of the hand For example, a straight line that represents the orientation of a surface.

[0106] For example, when calculating the height of a finger joint, the analysis unit 2060 calculates the height of each finger joint from the video frame 22. The analysis unit 2060 detects the finger joints and identifies their positions. The analysis unit 2060 then detects the key and identifies the reference line based on the key. The height of the finger joint is calculated based on the identified finger joint position and the reference line.

[0107] When calculating the magnitude of the angle between the joint lines, the analysis unit 2060 calculates from the video frame 22: Detect each finger joint and identify its position. Then, determine the position of these finger joints. For each pair of joint lines that share a common finger joint, The size of the angle between the joint lines is calculated.

[0108] When calculating the height of the wrist, the analysis unit 2060 calculates the position of the wrist from the video frame 22. The analysis unit 2060 also detects the joints that represent the target and identifies their positions. The reference line is identified by identifying the key on which the hand is placed. 0 calculates the wrist height based on the identified wrist position and the reference line.

[0109] When calculating the angle of the wrist, the analysis unit 2060 detects the hand and arm from the video frame 22. Then, the analysis section identifies the line representing the posture of the upper surface of the hand and the line representing the posture of the arm. The 2060 uses these lines to calculate the angle of the wrist.

[0110] When calculating the depth of the keystroke, the analysis unit 2060 calculates the depth of the keystroke from the video frame 22. Then, the analysis unit 2060 calculates the keystroke time for each detected key. In both the locked state and normal state, the key is turned on at the long side of the top surface of the key that is farthest from the camera 10. The analysis unit 2060 then calculates the line segments that represent the long sides of each key. The information about the normal state of a key is calculated by In the video frame 22 at the timing when the key is not in use, it is obtained from the line segment on the top side of the key, or In the video frame 22 at the time when the key is pressed, It can be obtained from the line segment on the top side of the key that is farther from the camera 10.

[0111] As mentioned above, the world coordinate system may be used to calculate each item value. In this case, the analysis unit 2060, as described above, analyzes the information (index) detected from the video frame 22. The item values ​​are calculated after converting the information (such as the position of the nodes) into information on the world coordinate system.

[0112] Here, in order to generate the user performance information 50 for the video frame 22, It is necessary to identify the performance time corresponding to the video frame 22. For example, as mentioned above, The time of the performance is relative to the time when the first performance (keystroke) is performed in the target piece of music. In this case, the analysis unit 2060 analyzes the video data 20 that is included in the acquired video data 20. The time point at which the video frame 22 representing the first performance is generated is determined as the reference time point. For other video frames 22, the analysis unit 2060 identifies the performance time. The time point at which the desired video frame 22 was generated is different from the time point at which the video frame 22 representing the reference time point was generated. Based on the difference and the frame rate of the camera 10, the video frame in which the performance point is to be identified is determined. The generation time of the program 22 is converted to the performance time.

[0113] <Generation of evaluation information 30: S108> The evaluation unit 2080 generates evaluation information 30 using the reference information 40 and the user performance information 50. Specifically, the evaluation unit 2080 evaluates the item information 44 included in the reference information 40 and the user Correspondence is established between the item information 54 included in the performance information 50 and the corresponding item information 44 and the item information 54. The corresponding item information 44 and the item information 54 are compared at the time of performance. These are information that match or are close to each other and should represent the same action (keystroke). Specifically, for example, the evaluation unit 2080 evaluates the item information 44 and the item information 54 that correspond to each other. For each evaluation item, an evaluation score is calculated that indicates the degree of agreement between the item values. The evaluation score calculation unit 2080 generates the evaluation information 30 using the calculated evaluation score. The method for calculating the core and the method for generating the evaluation information 30 using the evaluation score will be described later. do.

[0114] <<Processing flow>> FIG. 17 is a flowchart illustrating a process for calculating an evaluation score for each item of information 44. The loop process L1 is executed for each item information 44 included in the reference information 40. In S202, the evaluation unit 2080 evaluates all the item information 44. It is determined whether the loop process L1 has been executed for all the item information 44. If the loop processing L1 is executed as If there is one or more pieces of item information 44 that are not included in L1, the evaluation unit 2080 For example, the evaluation unit 2080 selects one of the items of information 44 from the list. The items are selected as the target of the loop process L1 in order from the earliest item information 44. The resulting item information 44 is referred to as item information 44-i.

[0115] The evaluation unit 2080 selects the item information 4 from the item information 54 included in the user performance information 50. For example, the evaluation unit 2080 identifies the item information 54 corresponding to the user 4-i (S204). Among the performance points 52 shown in the performance information 50, the performance corresponding to the item information 44-i is The evaluation unit 2080 then determines the closest performance time point 52. The item information 54 corresponding to the item information 44-i is identified as the item information 54 corresponding to the item information 44-i.

[0116] The evaluation unit 2080 compares the identified item information 54 with the item information 44-i to perform evaluation. An evaluation score is calculated for each item (S206). S208 is the end of the loop process L1. Therefore, the process in FIG. 17 proceeds to S202.

[0117] <<Evaluation score calculation method>> The evaluation score is calculated by dividing the item value 58 shown in the item information 54 by the item value 58 of the target evaluation item. The degree of agreement with the item value 48 shown in the item information 44 is shown. For example, for each evaluation item, An evaluation formula for calculating the evaluation score is prepared in advance. Regarding this, the value indicated by the item value 48 of the item information 44 and the value indicated by the item value 58 of the item information 54 are The evaluation formula then performs a predetermined calculation on the input value to obtain an evaluation score. Outputs the a.

[0118] For example, the evaluation formula can be the difference between item value 48 and item value 58, or the difference between item value 48 and item value 58. The difference or ratio may be standardized.

[0119] For example, the evaluation formula may be such that the allowable range determined based on the item value 48 includes the item value 58. For example, the acceptable range may be set to " ±20% of the value indicated by item value 48 (more than 0.8 times the value indicated by item value 48, 1.0 times the value indicated by item value 48) The evaluation formula takes item value 48 and item value 58. and determining whether the item value 58 is included in the acceptable range based on the item value 48. The evaluation formula is a value that represents the judgment result (for example, 1 if it is within the allowable range, 0 if it is not, etc.). If not, output 0.

[0120] In addition, for example, by defining multiple numerical ranges based on the item value 48, the user can The performance of 60 may be evaluated by a rank. For example, the number "value indicated by item value 48 ± 20%" Value range A, numerical range B, which is "outside numerical range A, within ±40% of the value indicated by item value 48," and Let's say that a numeric range C is defined as "outside of numeric range B." In this case, the item value 58 is in numeric range A, the highest rating is given, and if the item value 58 is in numeric range B, is the next highest rated item, and if the item value 58 is included in the numerical range C, it will be the lowest rated item. Therefore, a rank that indicates the degree of such evaluation is assigned to each numerical range. For numerical ranges with high ratings, a rank expressed as a smaller natural number is assigned. In the example, the ranges A to C are assigned the ranks 1, 2, and 3, respectively.

[0121] When evaluating by rank in this way, the evaluation formula takes item value 48 and item value 58, and Among the multiple numerical ranges, a numerical range that includes the item value 58 is identified. For example, in the example above, the item value If 58 is included in the numerical range B, the rank "2" is output.

[0122] <<Generation of evaluation information 30 using evaluation scores>> The evaluation unit 2080 uses the evaluation score to generate evaluation information 30. For example, the evaluation unit 2080 080 generates evaluation information 30 that indicates all the calculated evaluation scores. More specifically, The evaluation unit 2080 evaluates the timing of each keystroke (performance timing 42) in the target piece of music. The "evaluation items, The evaluation information 30 indicates a plurality of pairs of "evaluation score" and "evaluation score."

[0123] FIG. 18 is a diagram illustrating an example of the evaluation information 30 in a table format. The table 400 is represented as a table 400. The table 400 includes performance time 402 and evaluation result 403. 04. The evaluation result 404 shows a pair of evaluation items 406 and evaluation scores 408. The score 408 is a corresponding score for the performance of the user 60 at the corresponding performance point 402. The evaluation score for the evaluation item 406 is shown.

[0124] In addition, for example, the evaluation unit 2080 may calculate the evaluation score by tallying the evaluation scores in various ways. For example, the evaluation unit 2080 may generate the value information 30 for each performance time point. Calculate the statistical value (average value, etc.) of the evaluation score calculated for the item information 54 corresponding to the In this case, a score is obtained for each performance, which represents the overall evaluation of that performance. Hereafter, the statistical value of the evaluation score will be referred to as the statistical score. If different, the statistical score is calculated by taking into account the importance of each evaluation item (e.g., weighted average). may be.

[0125] FIG. 19 is a diagram illustrating an example of evaluation information 30 showing statistical scores for each performance point. The table shown in FIG. 5 is referred to as table 500. The table 500 is integrated with the performance time 502. The statistical score 504 is a pair of the total score 504 for the corresponding performance point 502. It is a statistical value of multiple evaluation scores calculated by

[0126] The evaluation scores may be tallied for each evaluation item. For each evaluation item, the evaluation scores calculated for each of the multiple performance points for that evaluation item In this case, the overall evaluation of the performance of the user 60 is shown for each evaluation item. will be done.

[0127] FIG. 20 is a diagram illustrating the evaluation information 30 showing the statistical scores for each evaluation item. The table shown in the figure is referred to as table 600. Table 600 is integrated with evaluation items 602. The statistical score 604 is a pair of scores for the corresponding evaluation item 602. It is a statistical value of multiple evaluation scores calculated by

[0128] In the example of Figure 20, the total for each evaluation score is calculated for the entire evaluation range. The aggregation for each evaluation score may be performed for each of multiple subranges included in the evaluation range. For example, a subrange is a section or a measure. When you want to aggregate by subrange, you need to use the criteria information 4. 0, including information that can identify each subrange (e.g., information that can identify a section). The evaluation unit 2080 identifies each partial range by referring to the reference information 40, and The cores are grouped into subranges. Then, the evaluation unit 2080 performs the following for each subrange: The evaluation scores are aggregated (for example, a statistical score is calculated).

[0129] FIG. 21 illustrates an example of evaluation information 30 showing statistical scores for each subrange and for each evaluation item. The table shown in FIG. 21 is referred to as table 700. The evaluation results 704 include a range 702 and an evaluation result 704. The evaluation result 704 includes evaluation items 706 and statistical scores. The statistical score 708 is for the performance time included in the corresponding subrange 702. Among the evaluation scores calculated for the evaluation item 706, the evaluation score is calculated from the evaluation score for the corresponding evaluation item 706. The resulting statistics are shown.

[0130] If the evaluation items can be classified into multiple groups, the evaluation scores are calculated based on the group of evaluation items. For example, the hand assessment items may be divided into two groups: one for the right hand and one for the left hand. The evaluation scores for the finger-related evaluation items can be categorized into thumb, index finger, and The evaluation scores can be classified into those for the thumb, middle finger, ring finger, and little finger.

[0131] FIG. 22 is a diagram illustrating the statistical values ​​of the evaluation scores tallied for each group. The table shown in the figure is called table 800. Table 800 contains the performance time 802 and the evaluation result. The evaluation result 804 includes an item group 806 and an evaluation score 808. The evaluation score 808 is calculated based on the performance at the corresponding performance point 802. The statistical values ​​of the evaluation scores for each evaluation item belonging to 806 are shown.

[0132] In this case, the evaluation scores are calculated for each group. Instead, statistical scores may be calculated for the entire evaluation range or for each subrange.

[0133] <Output of evaluation information 30> The performance evaluation device 2000 outputs the evaluation information 30 in an arbitrary manner. The device 2000 stores the evaluation information 30 in a storage device. 000 displays the evaluation information 30 on the display device. The device 2000 may transmit the evaluation information 30 to another device. For example, the performance evaluation device 20 00 is a server machine 80 that receives video data 20 from a user terminal 62 and operates accordingly. In this case (see FIG. 5(b)), the performance evaluation device 2000 transmits the evaluation information 30 to the user. The evaluation information 3 is then transmitted to the terminal 62. Then, the evaluation information 3 is transmitted to the user terminal 62 by various methods. 0 views are performed.

[0134] The performance evaluation device 2000 outputs the evaluation information 30 as it is, but outputs the evaluation information For example, the performance evaluation device 2000 may use the evaluation information 30 to The evaluation result screen is generated using the evaluation result information and displayed on the display device.

[0135] 23 is a diagram illustrating an example of an evaluation result screen in the first embodiment. Surface 900 shows a graph generated from the evaluation information 30. In this example, the statistical scores are In this example, the calculation is performed for each evaluation item. The time evolution of the total score is shown in a line graph.

[0136] In addition, in Figure 23, statistical scores that represent particularly poor evaluations are highlighted. The threshold value of the statistical score is shown by a dashed line, and statistical scores below this threshold are dotted. In the same way, statistical scores that represent particularly good evaluations are highlighted. Specifically, statistical scores greater than a predetermined threshold are highlighted. Make it possible.

[0137] The performance evaluation device 2000 performs a three-dimensional evaluation of the performance of the user 60 and the performance of the model performer. Generate a modeling image of the hand shape, output a screen containing the image, or output a video of the image The data may be stored in a storage device. Hand modeling of the playing of the user 60 The image is generated based on information such as the joint points of each finger and wrist obtained from the user performance information 50. Similarly, hand modeling video of the model performer can be used as reference information 40 It can be generated based on information such as the joint points of each finger and wrist obtained from the

[0138] The performance evaluation device 2000 preferably outputs evaluation information 30 together with the modeling video. Here, the evaluation results for each evaluation item are displayed in the modeling video. It is preferable that the item is displayed in association with a part related to the item. For example, The evaluation results for each finger joint are displayed near the corresponding wrist. The evaluation result is displayed near the corresponding finger.

[0139] By showing the evaluation information 30 along with the hand modeling image in this way, the user 60 Users can intuitively visually grasp the difference between their own performance and that of the model performer. 60 will be able to easily recognize areas that need to be corrected (practiced).

[0140] [Embodiment 2] <Summary> The performance evaluation device 2000 of the second embodiment, like the performance evaluation device 2000 of the first embodiment, The user 60 plays the target piece of music using the target instrument 70. The user 60 plays the target piece of music using the target instrument 70. The evaluation information 30 is generated by comparing the performance information 50 with the performance information 50. The evaluation of the performance in this section includes evaluation of the intensity or duration of the sound. The value is high when the strength of the keystrokes made by the user 60 is appropriate, The rating is low when the strength of the keystrokes made by the 60 is inappropriate. is high when the sound duration is appropriate and low when the sound duration is inappropriate.

[0141] The reference information 40 of the second embodiment is related to the corresponding music piece and includes information about the intensity or duration of the sound. To this end, the evaluation items of the standard information 40 include "sound intensity" and "sound duration." When an evaluation is made regarding sound intensity, the reference information 40 includes at least one of the following: For each note played, a standard of intensity is given. On the other hand, the duration of the note is evaluated. In this case, the standard information 40 indicates the standard length of each note to be played. The reference information 40 of the second embodiment is based on the various evaluation items (such as the height of the finger joints) described in the first embodiment. It may or may not include further criteria for:

[0142] The performance evaluation device 2000 of the second embodiment receives information representing a performance given by a user 60 and In the second embodiment, the user performance information 50 is The performance performed by the user 60 shows information about the strength or duration of each note. For this reason, the evaluation items of the user performance information 50 include at least "sound intensity" and "sound length." At least one of the following is included. When the evaluation of the sound intensity is performed, the user performance information 5 0 indicates information about the intensity of each sound. On the other hand, if an evaluation is made regarding the duration of the sound, The user performance information 50 indicates information about the duration of a note. 50 may further include information on the various evaluation items described in the first embodiment, It does not have to be included.

[0143] The performance evaluation device 2000 compares the reference information 40 with the user performance information 50. , and generates evaluation information 30. When evaluation is performed on the intensity of the sound, the performance evaluation device 20 00 compares the intensity of the corresponding sounds between the user performance information 50 and the reference information 40. By doing so, the intensity of the sound is evaluated. On the other hand, the duration of the sound is evaluated. In this case, the performance evaluation device 2000 determines whether the user performance information 50 and the reference information 40 are mutually An evaluation of the duration of the sound is made by comparing the duration of the corresponding sounds.

[0144] As will be described later, in this embodiment, the user performance information 50 is generated using a video The video data 20 may not be used to generate the user performance information 50. If not used, the camera 10 may not be provided.

[0145] <Example of effects> The inventions of Patent Documents 1 and 2 both include items for evaluating performance such as sound intensity and sound duration. In addition, the intensity and duration of the sound are not determined by the performer, as in Patent Documents 1 and 2. It is thought to be difficult to identify the victim's hand from the footage obtained by filming it from above.

[0146] In this regard, the performance evaluation device 2000 of this embodiment can evaluate the performance of the user 60. By comparing the user performance information 50 with the reference information 40, the intensity or duration of the sound is determined. Therefore, we can provide a new perspective on the performance of user 60. It is possible.

[0147] The performance evaluation device 2000 of this embodiment will be described in more detail below.

[0148] <Example of functional configuration> FIG. 24 is a block diagram illustrating the functional configuration of a performance evaluation device 2000 according to the second embodiment. In this example, the performance evaluation device 2000 includes a reference information acquisition unit 2020, user performance information The reference information acquisition unit 2020 acquires the reference information for the target song. The user performance information acquisition unit 2100 acquires the reference information 40 for the user performance information 50. The evaluation unit 2080 compares the reference information 40 with the user performance information 50 to obtain the As a result, evaluation information 30 is generated.

[0149] <Example of hardware configuration> The hardware configuration of the performance evaluation device 2000 of the second embodiment is the same as that of the performance evaluation device of the first embodiment. 2000, for example, is shown in FIG. 3. The storage device stores the functions of the performance evaluation device 2000 of the second embodiment in a computer 10. The program to make this happen is stored in 00.

[0150] <Example of Usage of Performance Evaluation Device 2000> The performance evaluation device 2000 of the second embodiment, like the performance evaluation device 2000 of the first embodiment, It can be used in the manner shown in Fig. 5. However, as described above, the usage manner described with reference to Fig. 5 The above is an example, and the specific usage of the performance evaluation device 2000 is not the same as the example explained with reference to FIG. Not limited.

[0151] <Processing flow> FIG. 25 illustrates the flow of processing executed by the performance evaluation device 2000 of the second embodiment. The reference information acquisition unit 2020 acquires the reference information 40 (S302 The user performance information acquisition unit 2100 acquires the user performance information 50 (S304). The unit 2080 generates evaluation information 30 using the reference information 40 and the user performance information 50 ( S306).

[0152] Note that S302 and S304 may be performed in the reverse order to that shown in FIG. They may be performed in parallel.

[0153] <Acquisition of standard information 40: S302> As described above, the reference information 40 in the second embodiment includes the "sound intensity" and the "sound length." The reference information 40 of the second embodiment is shown in FIG. Here, the criteria for the intensity of the sound are "time of performance, sound identification" and "time of The sound intensity can be expressed as a combination of "information, a value that represents the sound intensity." The reference information 40 indicating the reference for each note is, for example, the time when each note is played at the playing time 42. The score is shown, and the evaluation item 46 is the identification information of the evaluation item "sound intensity" and the identification of the sound played. The pair "information" is shown, and the value representing the intensity of the sound is shown in item value 48.

[0154] On the other hand, the standard for the duration of a note is "the time of performance, the identification information of the note, and the value that represents the duration of the note." Therefore, the standard information 4 that indicates the standard for the duration of the sound is For example, for each note, 0 indicates the time when it was played at the performance time 42, and the evaluation item 46 is " The evaluation item "note duration" is paired with the identification information of the played note. The value 48 indicates the length of the sound.

[0155] As will be described later, the performance evaluation device 2000 evaluates the strength and duration of the notes in relation to the fingering. The fingering evaluation is based on whether the fingering of the key is correct. Therefore, the reference information 40 is the basis for fingering. In this case, the reference information 40 may further indicate the reference for each note of the corresponding piece of music. The fingering criteria can be, for example, The performance time 42 indicates the time when the piece was played, and the evaluation item 46 indicates the identification of the evaluation item "fingering." The item value 48 indicates the fingering information to be used for playing. Furthermore, the strength of the sound, the length of the sound, or the fingering can be determined by the fingering in the first embodiment. It may also be evaluated in relation to the shape of

[0156] The reference information acquisition unit 2020 identifies a target song and acquires reference information 40 about the song. The method for identifying the target songs is the same as that described in the first embodiment. Can be used.

[0157] <<How to generate reference information 40>> For example, the reference information 40 is generated based on data recorded by a model performer. In this embodiment, the reference information 40 can be generated by, for example, The performance data is used to record the performance. The performance data includes the following for each note played: This includes information that can identify the time of performance, the identification information of the note, the intensity of the note, and the length of the note. An example of such performance data is MIDI (Musical Instrument Digital Interface) ce) data. In MIDI (registered trademark) terms, the information on the identification of the sound and the strength of the sound The magnitudes are called the "note number" and "velocity," respectively.

[0158] When generating the reference information 40 using performance data, the reference information generating device Then, the reference information generating device obtains the performance data of the For each note played, the time of playing, the note number of that note, and the velocity of that note are recorded. By extracting this information, a record of reference information 40 regarding sound intensity is generated. Similarly, the reference information generating device can generate information about each played note from the acquired performance data. By extracting information such as the time of performance, the note number, and the duration of the note, This allows a record of reference information 40 relating to the duration of a sound to be generated.

[0159] More specific examples of cases where MIDI (registered trademark) data is used as performance data In MIDI data, the output and stopping of a specific sound are A note-on message indicates that a sound is to be played, and a note-off message indicates that the sound is to be stopped. The intensity of the sound is expressed by the velocity indicated in the note-on message. The note duration can be specified by the time the note-on message is played. It can be identified by the difference between the point and the time when the note-off message is played.

[0160] In addition, for example, the reference information generating device may use a videotape recording of a performance by a model performer. Alternatively, the video data 20 may be acquired and the reference information 40 may be generated using the video data 20 . Here, one of the external characteristics of performance that affects the intensity of the sound is the keystroke speed. Therefore, the basic information generation device uses the video data By analyzing the data 20, the keystroke time (time of keystroke) of each model performer can be calculated. The time of playing, the identification information of the keyed note, and the keystroke speed are identified. Regarding the intensity of the sound, reference information 40 can be generated. It may be expressed at the start of the keystroke or at the end of the keystroke.

[0161] The keystroke speed when a certain key is pressed is, for example, the time between the start of the keystroke motion ts and the end of the keystroke motion. The start time of the keystroke can be expressed as the reciprocal of the difference between the end time te and the start time te, 1 / (te-ts). The video frame 22 representing the start of the key movement (the video frame 22 where the key begins to fall) is generated. Similarly, the end of a keystroke can be expressed as the video This represents the time when video frame 22 (the video frame with the key at its lowest) is generated. can be done.

[0162] Other external characteristics of playing that affect the intensity of the sound include, for example, the depth of keystrokes. Specifically, the deeper the key is pressed, the louder the sound. Some keys do not produce sound unless pressed to a certain depth. By analyzing the video data 20, for each keystroke performed by the model performer, The keystroke time (play time), the identification information of the keystroke, and the depth of the keystroke are identified. This allows the generation of reference information 40 regarding the intensity of each sound played. The time point may be represented as the start point of the keystroke or the end point of the keystroke. In addition, as explained in FIG. 13 of the first embodiment, the depth of the keystroke depends on the angle of the reference line and the distance between the reference points. It can be expressed using the length of a line segment, etc.

[0163] The reference information 40 regarding the sound intensity includes only one of the keystroke speed and the keystroke depth. may be shown, or both may be shown.

[0164] When the video data 20 is used to identify the duration of a sound, for example, the time when a key is pressed and The reference information generating device can identify the time difference between the time when the key was released and the time when the key was released. By analyzing the video data 20, the "keystrokes" of each keystroke performed by the model performer can be calculated. It identifies the key point (playing point), the identification information of the keyed note, and the point at which the key is released. This allows generating reference information 40 regarding the duration of each note played.

[0165] In order to be able to identify the pressed key from the video frame 22, for example, The key's identification information and the position of the image area representing that key on the video frame 22 are also stored. By referring to this information, the keystroke The identification information of the key can be determined from its position on the video frame 22.

[0166] The fingering is determined using the video data 20. Specifically, the reference information generating device , the pressed key and the finger that pressed the key are detected from the video frame 22, and However, the camera 1 that generates the video data 20 used to identify the fingering is 0 does not necessarily have to be installed so as to capture the performance of the user 60 from the side, It may be set up to capture the performance in a planar view (from above).

[0167] As in the first embodiment, the reference information 40 is generated based on the performances of a plurality of model performers. Information obtained from the performance may be utilized.

[0168] <Acquisition of user performance information 50: S304> The evaluation unit 2080 acquires the user performance information 50. The user performance information 50 has the following structure, for example: For example, the configuration of the reference information 40 is the same as that of the reference information 40 shown in FIG. 26. Here, the user performance information 50 is The music may be generated outside the performance evaluation device 2000 or may be generated by the performance evaluation device 2000. For example, the user performance information 50 may be generated internally by the user 6 When using performance data in which a performance by Performance data is acquired from outside the performance evaluation device 2000.

[0169] MIDI (registered trademark) data is used as performance data, and the performance is When an electronic musical instrument that can be recorded is used as the target musical instrument 70, the performance data of the user 60 is The performance data of the user 60 can be generated by the target instrument 70. It may also be generated based on recording data (WAV files, etc.) of a performance by a musician. Technology that converts audio data such as files into performance data such as MIDI (registered trademark) data. can utilize existing technology.

[0170] When the user performance information 50 is generated inside the performance evaluation device 2000, the analysis unit 20 For example, the analysis unit 2060 generates the user performance information 50. 0 performance data and extract necessary information from the performance data. The method of generating the reference information 50 is the same as the method of generating the reference information 40 from the performance data. This is similar to the method used.

[0171] In addition, for example, the analysis unit 2060 may analyze video data in which a performance by a user 60 is recorded. The video data 20 is acquired, and the user performance information 50 is generated by analyzing the video data 20. It is also possible to analyze the video data 20 in which the performance of the user 60 is recorded. The method for generating the performance information 50 is to decode the video data 20 in which the performance of the model performer is recorded. The method for generating the reference information 40 is the same as that for analyzing the reference information 40.

[0172] Here, both the performance data and the video data 20 are used to generate the user performance information 50. For example, the information on the intensity and duration of the sound in the user performance information 50 may be is generated from the performance data, and fingering information is generated from the video data 20.

[0173] <Generation of evaluation information 30: S306> The evaluation unit 2080 compares the reference information 40 with the user performance information 50 to evaluate the performance. The method for generating the evaluation information 30 is the same as in the first embodiment. For example, for the evaluation of sound intensity, the evaluation unit 2080 compares the corresponding item information 44 and By comparing with the item information 54, the evaluation score for the sound intensity for each sound is calculated. Similarly, for evaluation of the duration of a sound, the evaluation unit 2080 calculates the corresponding By comparing the item information 44 and the item information 54, the length of each sound is calculated. The evaluation score for the sound intensity is calculated. The formula and the evaluation formula used to calculate the evaluation score for the duration of the sound are prepared in advance. put.

[0174] The evaluation score for fingering is calculated based on the fingering shown in the reference information 40 and the user's performance information. This indicates the result of the judgment as to whether the fingerings shown in the information 50 are consistent with those shown in the information 50. For the item information 44 and the item information 54 corresponding to each other, the evaluation unit 2080 evaluates the item value 48. Whether the identification information of the finger shown matches the identification information of the finger shown in item value 58 For example, the evaluation score may be calculated based on the fingering. If the fingerings match, it shows a "1" and if the fingerings do not match, it shows a "0".

[0175] Also in this embodiment, as in the first embodiment, the evaluation unit 2080 For example, in this embodiment, the sound intensity may be calculated by summarizing the sound intensity. To tally the evaluation scores and note duration evaluation scores for each finger used in playing. Figure 27 shows the results of the analysis of the intensity and duration of the notes, based on the finger used for playing. FIG. 10 is a diagram illustrating an example of evaluation information 30 showing the statistical scores obtained.

[0176] <Output of evaluation information 30> The performance evaluation device 2000 outputs the evaluation information 30 in an arbitrary manner, as in the first embodiment. As explained in the first embodiment, the performance evaluation device 2000 stores the evaluation information 30 as it is. Instead of outputting the evaluation information 30 as is, the evaluation information 30 may be processed as appropriate and then output.

[0177] 28 is a diagram illustrating an example of an evaluation result screen in the second embodiment. Surface 1300 shows a graph generated from the evaluation information 30. In this example, the statistical scores are It is calculated for each finger used in the performance and for each evaluation item.

[0178] In this way, by aggregating the evaluation of the strength and duration of the sound for each finger used in playing, For example, it becomes easier to understand the habits of each hand and finger. For example, see the graph in Figure 28. In terms of both the intensity and duration of the notes, the score played using the left hand is Therefore, it is necessary to improve the technique of playing with the left hand. It is clear that this should be done.

[0179] For example, looking at the graph in Figure 28, the little finger on both the right and left hands You can see that the scores for performances using the same fingering are lower than the scores for performances using other fingers. Therefore, it is clear that the technique of playing with the little finger should be improved.

[0180] In the above embodiment, the program can be stored in various types of non-transitory computer-readable media. stored in a non-transitory computer readable medium and provided to a computer Non-transitory computer-readable media can include various types of tangible storage media. Examples of non-transitory computer-readable media include: magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives); Magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD- R, CD-R / W, semiconductor memory (e.g., mask ROM, PROM (Programmable ROM) ), EPROM (Erasable PROM), Flash ROM, RAM (Random Access Memory ) The program may also be stored on various types of temporary computer-readable media (transi The temporary data may be supplied to a computer via any computer-readable medium. Examples of computer-readable media include electrical signals, optical signals, and electromagnetic waves. The data-readable medium is a medium for transmitting a program via a wired communication path such as an electric wire or an optical fiber, or via a wireless communication path. It can supply RAM to the computer.

[0181] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. The configuration and details of the present invention are not intended to be understood by a person skilled in the art within the scope of the present invention. Various modifications can be made. [Explanation of symbols]

[0182] 10 Camera 20 Video Data 30 Evaluation Information 40 Standard Information 42 Performance time 44 item information 46 Evaluation Items 48 Item Values 50 User performance information 52 Performance time 54 item information 56 Evaluation Items 58 Item Values 60 users 62 User terminals 70 Target Instruments 80 Server Machine 110 fingers 120 keys 130 Reference Line 140 moves 142 fingers 150 Reference Line 160 keys 170 straight line 180 straight line 190 keys 200 line segments 210 line segments 300 Song selection screen 310 Search Area 320 Song selection area 330 Evaluation range designation area 400 tables 402 Performance time 404 Evaluation Results 406 Evaluation Items 408 rating score 500 tables 502 Performance time 504 Statistical Score 600 tables 602 evaluation items 604 Statistical Score 700 tables 702 Subrange 704 Evaluation Results 706 evaluation items 708 Statistical Score 800 tables 802 Performance time 804 Evaluation Results 806 Item Group 808 rating score 900 Evaluation result screen 1000 computers 1020 Bus 1040 processor 1060 memory 1080 storage device 1100 Input / Output Interface 1120 Network Interface 1300 Evaluation result screen 2000 Performance Evaluation Device 2020 Standard Information Acquisition Department 2040 Video Data Acquisition Unit 2060 Analysis Department 2080 Evaluation Department 2100 User performance information acquisition unit

Claims

1. On the computer, a reference information acquisition process for acquiring reference information indicating reference item values ​​for evaluation items related to musical instrument performance in a piece of music performed by a user; a video data acquisition process for acquiring video data obtained by capturing an image of the user playing the musical piece on the musical instrument with a camera installed at a position and posture that allows the image to be captured from the side; an analysis process of generating user performance information indicating item values ​​of the evaluation items for the performance of the musical piece by the user by analyzing images of the user's hands included in the video data, detecting the joints of the fingers of the user's hands from video frames included in the video data, and calculating the item values ​​of the evaluation items based on the size of an angle formed by two joint lines that have a common end point at the positions of the detected joints and that connect two adjacent joint points; and an evaluation process for generating evaluation information representing an evaluation of the performance by the user by comparing the reference information with the user's performance information.

2. the reference information includes height information determined from the positional relationship between the hand joint points and a predetermined reference plane; the user performance information includes height information determined from a positional relationship between a hand joint point and the predetermined reference plane when the user plays the piece of music, The program according to claim 1 , wherein the evaluation process generates the evaluation information including an evaluation of a height determined from a positional relationship between the joint points of the hand and the predetermined reference plane.

3. a reference information acquisition unit that acquires reference information indicating reference item values ​​for evaluation items related to musical instrument performance in a piece of music performed by a user; a video data acquisition unit that acquires video data obtained by capturing an image of the user playing the musical piece on the musical instrument with a camera that is installed in a position and posture that allows the image to be captured from the side; an analysis unit that generates user performance information indicating item values ​​of the evaluation items for the performance of the musical piece by the user by analyzing images of the user's hands included in the video data, detects joints of the fingers of the user's hands from video frames included in the video data, and calculates the item values ​​of the evaluation items based on the size of an angle formed by two joint lines that have a common end point at the positions of the detected joints and that connect two adjacent joint points; and an evaluation unit that generates evaluation information representing an evaluation of the performance by the user by comparing the reference information with the user's performance information.

4. A computer-implemented performance evaluation method, comprising: a reference information acquisition step of acquiring reference information indicating reference item values ​​for evaluation items related to musical instrument performance in a piece of music performed by a user; a video data acquisition step of acquiring video data obtained by capturing an image of the user playing the musical piece on the musical instrument with a camera installed at a position and posture that allows the image to be captured from the side; an analysis step of generating user performance information indicating item values ​​of the evaluation items for the performance of the musical piece by the user by analyzing images of the user's hands included in the video data, detecting joints of the fingers of the user's hands from video frames included in the video data, and calculating the item values ​​of the evaluation items based on the size of an angle formed by two joint lines that have a common end point at the positions of the detected joints and that connect two adjacent joint points; and an evaluation step of generating evaluation information representing an evaluation of the performance by the user by comparing the reference information with the user's performance information.

Images