Information processing apparatus, information processing method, and program

Abstract

To provide an information processor, an information processing method, and a program capable of efficiently improving a skill level of drum performance.SOLUTION: According to an embodiment of the present disclosure, there is provided an information processing apparatus including a first acquisition unit and an evaluation unit. The first acquisition unit acquires an image of a drum performance by a performer. The evaluation unit evaluates a proficiency level related to the drum performance based on the image acquired by the first acquisition unit. This makes it possible to efficiently improve the level of proficiency in drum performance.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program that can be used to improve proficiency in playing musical instruments. [Background technology]

[0002] Patent Document 1 describes an information processing device that aims to efficiently assist in the acquisition of performance skills such as piano playing. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Document 1: International Publication No. 2020 / 100671 Summary of the Invention [Problem to be solved by the invention]

[0004] Thus, there is a demand for technology that can efficiently improve the proficiency of musical instrument playing.

[0005] In view of the above circumstances, one object of the present invention is to provide an information processing device, an information processing method, and a program that enable efficient improvement in drum playing proficiency. [Means for solving the problem]

[0006] In order to achieve the above object, an information processing device according to an embodiment of the present invention includes a first acquisition unit and an evaluation unit. The first acquisition unit acquires an image of a drum performance by a performer. The evaluation unit evaluates the level of proficiency in playing the drums based on the image acquired by the first acquisition unit.

[0007] An information processing method according to one aspect of the present invention is an information processing method executed by a computer system, and includes acquiring an image of a drum performance by a performer. The level of proficiency in playing the drums is evaluated based on the acquired images.

[0008] A program according to one aspect of the present invention causes a computer system to acquire an image of a drum performance by a player, and evaluate the player's level of proficiency in drum performance based on the acquired image. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a schematic diagram illustrating an example of the basic configuration of a drum practice assistance system according to an embodiment. [Figure 2] 10 is a flowchart illustrating an example of a basic operation of the information processing device. [Figure 3] FIG. 10 is a schematic diagram showing an example of calculating a score using a learning model. [Figure 4] FIG. 1 is a schematic diagram for explaining learning of a learning model using teacher data. [Figure 5] FIG. 10 is a schematic diagram showing an example of score calculation using a learning model that uses extracted information extracted from an image of a drum performance as input. [Figure 6] FIG. 1 is a schematic diagram for explaining learning of a learning model using teacher data. [Figure 7] FIG. 10 is a schematic diagram showing an example of score calculation using a learning model that inputs both an image of a drum performance and extracted information. [Figure 8] FIG. 1 is a schematic diagram for explaining learning of a learning model using teacher data. [Figure 9] FIG. 10 is a schematic diagram showing proficiency evaluation by rule-based processing. [Figure 10] FIG. 10 is a schematic diagram showing proficiency evaluation by rule-based processing. [Figure 11] FIG. 10 is a schematic diagram showing proficiency evaluation by rule-based processing. [Figure 12]FIG. 10 is a block diagram showing another example of the functional configuration of the information processing device. [Figure 13] 10 is a table showing an example of one or more evaluation items related to drum performance. [Figure 14] 10 is a table showing an example of one or more evaluation items related to drum performance. [Figure 15] 10 is a table showing an example of one or more evaluation items related to drum performance. [Figure 16] 10 is a schematic diagram showing an example of an output of evaluation results and support information. FIG. [Figure 17] 10 is a schematic diagram showing an example of an output of evaluation results and support information. FIG. [Figure 18] FIG. 1 is a block diagram illustrating an example of the hardware configuration of a computer that can be used as an information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0011] [Drum practice support system] FIG. 1 is a schematic diagram showing an example of the basic configuration of a drum practice support system according to one embodiment of the present invention. The drum practice support system 1 supports the practice of a drummer 2 who plays the drums, and is capable of efficiently improving the level of proficiency in drumming. The level of proficiency refers to the degree of proficiency (degree of proficiency). Drum practice support system 1 can also be called a drum proficiency improvement system. Player 2 can also be considered a user of this drum practice support system.

[0012] In this disclosure, drumming includes any method of playing performed on any form of drum (taiko). In this embodiment, as shown in FIG. 1, a drum set 4 is played using sticks 3. 1 shows a drum set 4 including a bass drum 5, a snare drum 6, a high tom 7, a low tom 8, a floor tom 9, a high-hat cymbal 10, a crash cymbal 11, and a ride cymbal 12. The drum set 4 is not limited to this configuration, and any configuration may be used. The present technology can be applied to drum performances that are not limited to the drum set 4, but also include performances of Japanese drums using drumsticks and various percussion instruments using hands. Of course, performances using electronic instruments such as electronic drums are also included in drum performances. Furthermore, instruments that are not classified as drums when viewed individually, such as the various cymbals included in the drum set 4 shown in Figure 1, are often placed and played together with the drums. For example, when playing Japanese drums, gongs and other instruments may be played together. When playing percussion, wind chimes, cowbells, shakers, etc. may also be played together. Electronic drum sets may also include electronic cymbals. In this disclosure, playing other types of instruments that are played along with drums is also considered to be included in drum performance. That is, playing various cymbals such as those shown in FIG. 1 is also included in drum performance, and the present technology can be applied to this. There are also performers who play by hitting objects that are not manufactured as musical instruments, such as pots and tables. These acts of hitting non-musical objects are also included in drumming. Furthermore, drumming practice may be performed using a dedicated practice stand (such as a practice pad) or a desk, etc. In this disclosure, such drumming practice itself is also considered to be included in drumming.

[0013] As shown in FIG. 1, drum practice support system 1 has imaging device (camera) 14 and information processing device 15. The imaging device 14 and the information processing device 15 are connected to each other via wire or wirelessly so that they can communicate with each other. The connection form between the devices is not limited, and it is possible to use, for example, wireless LAN communication such as WiFi or short-range wireless communication such as Bluetooth (registered trademark).

[0014] The image capturing device 14 is positioned so that it can capture an image of the drum performance by the performer 2. It will be placed. As the imaging device 14, for example, a digital camera equipped with an image sensor such as a CMOS (Complementary Metal-Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor is used. Any other imaging device capable of capturing images of a drum performance may be used. In this disclosure, images include both still images and moving images (videos).

[0015] The information processing device 15 has hardware necessary for configuring a computer, such as a processor such as a CPU, GPU, or DSP, a memory such as a ROM or RAM, and a storage device such as an HDD (see FIG. 18). For example, a CPU loads a program according to the present technology, which is pre-recorded in a ROM or the like, into a RAM and executes the program, thereby executing an information processing method according to the present technology. For example, the information processing device 15 can be realized by any computer such as a PC (Personal Computer). Of course, hardware such as FPGA and ASIC may also be used.

[0016] In this embodiment, a CPU or the like executes a predetermined program to configure a first acquisition unit 16 and an evaluation unit 17 as functional blocks. Of course, dedicated hardware such as an IC (integrated circuit) may be used to realize the functional blocks. The program is installed in the information processing device 15 via, for example, various recording media. Alternatively, the program may be installed via the Internet or the like. The type of recording medium on which the program is recorded is not limited, and any computer-readable recording medium may be used. For example, any computer-readable non-transitory storage medium may be used.

[0017] FIG. 2 is a flowchart showing an example of the basic operation of the information processing device 15. The first acquisition unit 16 acquires an image of the drum performance by the player 2 (step 101). The evaluation unit 17 evaluates the level of proficiency in drum playing based on the image of the drum playing acquired by the first acquisition unit 16. For example, as shown in FIG. 1, a comprehensive evaluation of the drum performance is carried out, and a score indicating the level of proficiency (evaluation score) is calculated as the evaluation result.

[0018] In the example shown in Fig. 1, a five-point scale from A to E is used, and one of A to E is calculated as the score. The score is calculated as a score (points) ranging from 0 to 100 points. Of course, the evaluation of proficiency is not limited to a comprehensive evaluation of drum performance. It is possible to evaluate proficiency with respect to various evaluation items related to drum performance. That is, the evaluation unit 17 can perform evaluation with respect to one or more evaluation items related to drum performance. This makes it possible to evaluate drum performance proficiency with high accuracy. Furthermore, the calculation of the score is not limited to the graded evaluation or the allocation of points as exemplified in FIG. 1, and the score may be calculated in any manner. Furthermore, evaluation methods other than score calculation may be adopted to evaluate proficiency. For example, any evaluation method may be implemented, such as displaying evaluation comments or outputting a specific voice. For example, it is also possible to implement an overall evaluation or evaluation of one or more evaluation items without outputting proficiency or evaluation scores as parameters. The evaluation comments are included in the support information for improving the level of proficiency.

[0019] 1, the first acquisition unit 16 corresponds to an embodiment of the first acquisition unit according to the present technology, and the evaluation unit 17 corresponds to an embodiment of the evaluation unit according to the present technology.

[0020] 1, imaging device 14 and information processing device 15 are separately prepared to construct drum practice support system 1. However, this is not limiting, and any computer with imaging capabilities may be used to construct drum practice support system 1. That is, a device in which the imaging device 14 and the information processing device 15 shown in FIG. 1 are integrated may be used as an embodiment of the information processing device according to the present technology. Examples of computers with imaging capabilities that can be used include smartphones, tablet devices, HMDs (Head Mounted Displays) such as AR (Augmented Reality) glasses and VR (Virtual Reality) glasses, and PCs.

[0021] [Proficiency assessment (processing using machine learning)] There are no limitations on the method by which the evaluation unit 17 evaluates the level of proficiency in drum playing, and any technique (algorithm, etc.) may be used. For example, any machine learning algorithm may be used, such as a DNN (Deep Neural Network), an RNN (Recurrent Neural Network), or a CNN (Convolutional Neural Network). For example, by using AI (artificial intelligence) that performs deep learning, it becomes possible to calculate scores with high accuracy for various evaluation items. The following describes a case where proficiency is evaluated by a process using machine learning.

[0022] FIG. 3 is a schematic diagram showing an example of score calculation using a learning model. In the example shown in FIG. 3, an image 19 of a drum performance is used as input, and machine learning is performed to estimate a score indicating the proficiency level of drum performance. That is, by inputting image 19 of a drum performance into a trained learning model 20 that has undergone machine learning to estimate proficiency, a process is executed to obtain a score indicating proficiency from learning model 20. This process makes it possible to evaluate proficiency with high accuracy. Of course, it is not limited to calculating scores, and processing using a learning model can be applied to any evaluation method. In other words, any machine learning model trained to estimate evaluations can be used to perform an overall evaluation or an evaluation for one or more items. The learning model 20 can also be called a machine learning model 20 or a trained model 20.

[0023] FIG. 4 is a schematic diagram for explaining learning of the learning model 20 using teacher data. 4, for learning of the learning model 20, training data in which training data is associated with training labels is input to the learning unit 21. The training data is data for training the learning model 20, which estimates a correct answer for an input. 4, in this embodiment, drum performance image 22 is input to learning unit 21 as learning data. In addition, a score indicating proficiency is input to learning unit 21 as teacher label 23. Teacher label 23 is a correct answer (correct answer data) corresponding to drum performance image 22 for learning.

[0024] In this embodiment, data in which scores (teaching labels 23) are associated with learning drum performance images 22 (learning data) is used as training data. Therefore, the learning model 20 is a prediction model that has been machine-learned using the drum performance images 22 and scores indicating proficiency as training data. There are no limitations on the method for creating the training data (dataset of drum performance images 22 and scores for learning). For example, the training data may be created manually. Alternatively, training data created in advance may be acquired and input to the learning unit 21. The scores (labels 23) used as training data are associated with scores for various evaluation items related to drum performance, and the learning model 20 is trained. This makes it possible to perform evaluations for each evaluation item through processing using machine learning. In other words, it becomes possible to obtain scores for each evaluation item from the learning model 20.

[0025] As shown in Fig. 4, the learning unit 21 uses teacher data and performs learning based on a machine learning algorithm. Through learning, parameters (coefficients) for calculating a correct answer (teacher label) are updated and generated as learned parameters. A program incorporating the generated learned parameters is generated as a learning model 20.

[0026] The backpropagation method is one example of a learning method used for learning models. Backpropagation is a commonly used learning method for training neural networks. A neural network is a model that originally imitates the neural circuits of the human brain, and has a layered structure consisting of three types of layers: an input layer, an intermediate layer (hidden layer), and an output layer. Neural networks with many hidden layers are called deep neural networks, and the deep learning techniques used to train them are known as models that can learn complex patterns hidden in large amounts of data. Backpropagation is one such learning method, and is often used, for example, in training CNNs, which are used to recognize images and videos. Furthermore, as a hardware structure for realizing such machine learning, a neurochip / neuromorphic chip incorporating the concept of neural networks can be used.

[0027] The algorithm for training the learning model 20 is not limited, and any machine learning algorithm may be used. For example, machine learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, inverse reinforcement learning, active learning, and transfer learning. Supervised learning learns features based on given labeled training data (teacher data), which makes it possible to derive labels for unknown data. Unsupervised learning analyzes large amounts of unlabeled training data to extract features, and then performs clustering based on the extracted features. This makes it possible to analyze trends and make future predictions based on vast amounts of unknown data. Semi-supervised learning is a method that combines supervised learning and unsupervised learning. After learning features through supervised learning, the system is given a huge amount of training data through unsupervised learning, and the system repeats the learning process while automatically calculating the features. Reinforcement learning deals with the problem of an agent in an environment observing its current state and deciding what action to take. The agent learns rewards from the environment by selecting actions, and learns a strategy that will maximize rewards through a series of actions. In this way, by learning the optimal solution in a certain environment, it is possible to reproduce human judgment and even enable a computer to acquire judgment skills that surpass those of humans. Furthermore, machine learning models such as HMM (Hidden Markov Model) and SVM (Support Vector Machine) may also be used.

[0028] The learning model 20 generated by the learning unit 21 is incorporated into the evaluation unit 17 shown in Fig. 1. Then, the evaluation unit 17 estimates the score. The learning unit 21 shown in FIG. 4 may be included in the information processing device 15, and the information processing device 15 may execute learning of the learning model 20. On the other hand, the learning unit 21 may be configured outside the information processing device 15. That is, learning may be performed in advance by the learning unit 21 outside the information processing device 15, and only the learned learning model 20 may be incorporated into the evaluation unit 17. Furthermore, the specific configuration of the learning unit 21 and the learning unit 21 for learning the learning model 20 is not limited.

[0029] The application of a machine learning algorithm may be performed for any process within the present disclosure, i.e., any process described within the present disclosure may be subjected to a process using machine learning.

[0030] [Evaluating proficiency using extracted information] It is also possible to evaluate the proficiency of drumming using information that can be extracted from the image 19 of the drumming performance. The extracted information may include, for example, one or more feature points related to the drum performance, skeletal information of the player 2, the center of gravity of the player 2, the player's facial expression, and the movement of the sticks 3 used in playing the drums.

[0031] For one or more feature points related to a drum performance, specific parts such as the player 2, the drum set 4, and the sticks 3 are defined as feature points, and the position, movement (momentum), speed, acceleration, etc. of the feature points are extracted as extracted information. The parts defined as feature points may be set arbitrarily. Furthermore, a specific coordinate system may be set arbitrarily to detect position information. For example, it is possible to obtain the position, movement (momentum), speed, acceleration, etc. of each part of the player 2 based on the skeletal information of the player 2. It is also possible to obtain the position, movement (momentum), movement, speed, acceleration, etc. of the center of gravity of the player 2 based on the center of gravity of the player 2. Note that information regarding the center of gravity of the player 2 can also be obtained from the skeletal information. Based on the performer's facial expression, it is possible to obtain information such as whether the performer is smiling, frowning, or showing a relaxed expression. Based on the movement of the stick 3 used in playing the drums, it is possible to obtain the position, movement (momentum), speed, acceleration, etc. of each part of the stick 3. It is also possible to obtain information such as the speed of the stick 3 before the attack, the speed of the stick 3 after the attack, and the difference in speed before and after the attack. Note that an attack refers to the moment when the stick 3 hits the drum and produces sound. Any other information that can be extracted from the image 19 of the drum performance may be used as the extracted information.

[0032] There is no limitation on the method for extracting extracted information from the drum performance image 19, and any technique (algorithm) may be used. For example, any image recognition technology such as matching processing using a model image of an object, edge detection, projective transformation, etc. may be used. Skeleton estimation (bone estimation) may also be used. Furthermore, an externally constructed library with existing image processing or machine learning functions may also be used. Any machine learning algorithm may be used to extract the extracted information, for example, performing semantic segmentation on the image information to determine the object type for each pixel in the image. By using the extracted information, it is possible to evaluate the proficiency of drumming with high accuracy.

[0033] FIG. 5 is a schematic diagram showing an example of score calculation using a learning model that uses extracted information extracted from an image 19 of a drum performance as an input. As shown in FIG. 5, by performing machine learning using extracted information extracted from image 19 of a drum performance as input, it is possible to estimate a score indicating the proficiency level of drumming. In this case, extracted information extracted from drum performance image 19 is input to a trained learning model 24 that has undergone machine learning to estimate proficiency, and a process is executed to obtain a score indicating proficiency from learning model 28. This makes it possible to evaluate proficiency with high accuracy.

[0034] FIG. 6 is a schematic diagram for explaining the learning of the learning model 24 using the training data. In this embodiment, extracted information 25 extracted from learning drum performance image 19 is used as learning data. Data in which this learning data is associated with a score (teaching label 26) is used as teaching data. Therefore, the learning model 24 is a prediction model that has been machine-learned using the extracted information 25 extracted from the image of the drum performance and the score indicating the proficiency level as training data. 6, the learning unit 27 uses the training data and executes learning based on a machine learning algorithm, thereby generating a learning model 24. The algorithm for training the learning model 24 is not limited, and any machine learning algorithm may be used.

[0035] Scores for various evaluation items related to drum performance are associated as scores (labels 26) used as training data, and the learning model 24 is trained. This makes it possible to perform evaluations for each evaluation item using machine learning processing. In other words, it becomes possible to obtain scores for each evaluation item from the learning model 24. In this case, by using extracted information related to the evaluation items, it is possible to improve the accuracy of estimating the score.

[0036] FIG. 7 is a schematic diagram showing an example of score calculation using a learning model that inputs both the drum performance image 19 and the extracted information. As shown in Figure 7, by performing machine learning using both an image 19 of a drum performance and information extracted from the image 19 as input, it is possible to estimate a score indicating the level of proficiency in drumming. In this case, by inputting both the image 19 of the drum performance and the extracted information extracted from the image 19 into a trained learning model 28 that has undergone machine learning to estimate proficiency, a process is executed to obtain a score indicating proficiency from the learning model 28. This makes it possible to evaluate proficiency with high accuracy.

[0037] FIG. 8 is a schematic diagram for explaining the learning of the learning model 28 using the training data. In this embodiment, a set of learning drum performance images 29 and extracted information 30 extracted from learning drum performance images 29 is used as learning data. Data in which a score (teaching label 31) is associated with this learning data is used as teaching data. Therefore, the learning model 28 is a prediction model that has been machine-learned using a set of the drum performance image 29, the extracted information 30 extracted from the drum performance image 29, and a score indicating the proficiency level as training data. 8, the learning unit 32 uses the training data and executes learning based on a machine learning algorithm, thereby generating a learning model 28. The algorithm for training the learning model 28 is not limited, and any machine learning algorithm may be used.

[0038] Scores for various evaluation items related to drum performance are associated as scores (labels 31) used as training data, and the learning model 28 is trained. This makes it possible to perform evaluations for each evaluation item using machine learning processing. In other words, it becomes possible to obtain scores for each evaluation item from the learning model 28. In this case, by using extracted information related to the evaluation items, it is possible to improve the accuracy of estimating the score.

[0039] [Proficiency assessment (rule-based processing)] 9 to 11 are schematic diagrams showing evaluation of proficiency by processing using a rule base. As shown in FIGS. 9 to 11, the evaluation unit 17 can also evaluate the proficiency level in drum playing by processing using a rule base.

[0040] In the example shown in FIG. 9, an image 19 of a drum performance is input, and processing using a rule-based algorithm is executed to calculate a score indicating the proficiency level of the drum performance. In the example shown in FIG. 10, extracted information extracted from image 19 of a drum performance is used as input, and processing using a rule-based algorithm is executed to calculate a score indicating the proficiency level of drum performance. In the example shown in Figure 11, both image 19 of a drum performance and extracted information extracted from image 19 are used as input, and processing is performed using a rule-based algorithm to calculate a score indicating the drum performance proficiency.

[0041] In this way, it is possible to perform an evaluation of drumming proficiency through rule-based processing using at least one of the image 19 of the drumming performance and the extracted information extracted from the image 19. Of course, it is also possible to calculate scores for various evaluation items related to the drumming performance. There are no particular limitations on the specific algorithms and the like that are executed as rule-based processing. Any rule-based algorithm, such as matching technology, image recognition technology, or analysis technology, may be used.

[0042] [Evaluating proficiency using detection information / auxiliary information] In order to evaluate the proficiency of drumming, information other than the image 19 of the drumming performance and the extracted information extracted from the image 19 of the drumming performance may be used. Examples of the other information include detection information detected in response to drum playing, auxiliary information related to drum playing, and the like.

[0043] The detected information includes any information detected in response to the drum performance by the performer 2. The detected information is typically information detected by a detection device other than the imaging device 14. Examples of detection devices include a microphone, a computer capable of importing and processing performance data such as MIDI (registered trademark) (Musical Instrument Digital Interface) data, a gravity center meter, and a distance measurement sensor. The detection device may be any of various wearable devices that can be worn by the player 2 and various sensors that can be mounted on the wearable devices. For example, an IMU (Inertial Measurement Unit) sensor, a GPS sensor, a temperature sensor, or other biosensor may be used. Alternatively, an IMU sensor or the like may be attached to the stick 3 as a detection device. The various pieces of information detected by these various detection devices, sensors, etc. may be used to evaluate the level of proficiency in drum playing.

[0044] As the detected information, for example, sound information, performance time, performance tempo, sound interval, movement of the performer, center of gravity of the performer, or physical condition of the performer can be detected. The sound information is detected as audio data by a microphone or as MIDI (registered trademark) data. Note that sound information of other instruments being played can also be detected as detected information. The performance time is detected as, for example, the time from the start to the end of the performance. The performance tempo is detected as, for example, BPM (Beats Per Minute). The sounding interval is detected as the interval between sounds produced in response to a performance. For example, the sounding interval is detected for each part of the performer 2. For example, a performance may involve the right hand striking the hi-hat cymbal 10 in sixteenth notes while the left hand striking the snare drum 6 in quarter notes. In this case, the sounding intervals of the right hand and the left hand may be detected separately. That is, information is detected that indicates that the sounding intervals of the right hand are relatively short and that of the left hand are relatively long. The sound generation interval for each part may be the average value within a predetermined playing time, or the sound generation interval that occurs most frequently statistically may be used. The sound generation interval of each part can be considered the interval between sounds generated by each part, and can also be considered information indicating the speed of the performance action of each part.

[0045] The movement of the performer is detected, for example, from the position information and displacement of each part based on a predetermined coordinate system. For example, the acceleration, speed, and momentum of each part can be detected. The center of gravity of the player is detected, for example, from position information based on a predetermined coordinate system. The physical state of the performer is detected by, for example, a biosensor. For example, muscle relaxation / tension and other conditions can be detected as detected information.

[0046] It is possible that the same type of information as the extracted information extracted from image 19 of a drum performance is acquired as detected information. For example, the movement of the performer based on the skeletal information of the performer is acquired as extracted information. On the other hand, the movement of the performer is acquired as detected information detected by a wearable device worn by performer 2. Such a case is also possible. By using the detection information, it becomes possible to calculate scores with high accuracy for various evaluation items.

[0047] The auxiliary information includes various information that assists the player 2 in playing the drums. For example, the auxiliary information may include information on correct performance, information on past performances, and information on other performance sounds. The correct performance information includes information that can teach how the piece should be played, such as information about the musical score of the piece to be played and information about the piece in which each drum, cymbal, etc. included in the drum set 4 is sounding at the correct timing. The correct tempo (BPM) and click (metronome) information indicating the correct tempo may also be included. Furthermore, an image of a drum performance by a highly skilled performer who is playing accurately and can serve as a model may also be used as the correct information. The past performance information includes performance information of the same piece of music previously performed by performer 2 or another performer. The information on other performance sounds includes the performance sounds of other instruments (parts) that are being played together, such as correct information on the performance of other instruments, real-time performance information of other instruments that are being played together, and past performance information of other instruments.

[0048] By using auxiliary information, it becomes possible to calculate scores with high accuracy for various evaluation items. For example, by comparing the drum performance of player 2 with the performance accuracy information, it is possible to evaluate the level of proficiency in drum performance. Also, by comparing with past performance information, it is possible to evaluate the level of improvement in drum performance. It should be noted that the same type of information as the extracted information or detected information may also be used as auxiliary information.

[0049] FIG. 12 is a block diagram showing another example of the functional configuration of the information processing device 15. As shown in Figure 12, when detection information and auxiliary information are used to evaluate the proficiency of drum playing, a second acquisition unit 40 that acquires at least one of the detection information and auxiliary information is configured as a functional block in the information processing device 15. The second acquisition unit 40 is configured, for example, by a CPU or the like executing a predetermined program, similar to the first acquisition unit 16 and the evaluation unit 17. Of course, to realize the second acquisition unit 40, dedicated hardware such as an IC (integrated circuit) may be used.

[0050] [Evaluation items for drum performance] 13 to 15 are tables showing examples of one or more evaluation items related to drum performance. As shown in FIGS. 13 to 15, the evaluation items include evaluation items relating to the sound of the performance, evaluation items relating to the movement of the sticks 3 used in playing the drums, and evaluation items relating to the movement of the player 2. The evaluation unit 17 can calculate a score indicating the level of proficiency for each of these evaluation items by performing processing using machine learning or rule-based processing, thereby enabling the level of proficiency in drumming to be evaluated with high accuracy. In the example shown in Figures 3A to 3C, each evaluation item is evaluated on a five-point scale from A to E, and a score (points) ranging from 0 to 100 is calculated. Of course, the score calculation method and evaluation method are not limited.

[0051] FIG. 13 shows an example of evaluation items related to performance sounds. Evaluation items related to the sounds being played include, for example, control of sound timing, control of sound dynamics, control of timbre, stability of repeated playing, and whether or not there is communication with other played sounds. A score can be calculated for these evaluation items.

[0052] (Controlling the timing of pronunciation) It is possible to evaluate how well the performer 2 controls the timing of sound production. In Fig. 13, whether or not sound production is performed at the intended timing is listed as an evaluation item. For example, evaluation of this evaluation item is carried out at parts where the rhythm changes in the song structure, such as the transition from the A melody to the B melody, the part where the song enters the chorus, etc. Of course, evaluation is not limited to parts where the rhythm changes. For example, based on image 19 of a drum performance, it is possible to evaluate the control of sound timing from the viewpoint of whether or not successive hits are evenly performed and whether or not the timing of attacks varies. The extracted information may include, for example, information about the skeleton of the performer 2 and information about the movement of the sticks 3, and the like, which can be used for evaluation of this evaluation item. As the detection information, sound information (sound information) can be used. For example, when an electronic drum performance is being performed, MIDI (registered trademark) data can be used. Of course, normal drum sounds can also be acquired using a microphone or the like. Furthermore, the performance tempo, the movement of the performer, and the like can also be used as the detection information. As auxiliary information, it is possible to use correct performance information. For example, MIDI (registered trademark) data, musical scores, performance tempo, images of model drum performances, and other information can be used as correct performance information to evaluate this evaluation item. In addition, the information, parameters, etc. used to evaluate the control of sound generation timing are not limited and may be set arbitrarily.

[0053] The available extracted information, detected information, and auxiliary information are merely examples, and other types of extracted information, detected information, and auxiliary information may be used, as well as other evaluation items described below.

[0054] (Controlling the dynamics of sound) It is possible to evaluate how well performer 2 controls the dynamics of the sound. In Figure 13, whether or not the intended dynamics can be produced is listed as an evaluation item. For example, it is possible to evaluate how well the dynamics of a sound can be controlled within the same instrument. It is also possible to evaluate how well the dynamics of a sound can be controlled in relation to other instruments (balance). Of course, it is also possible to evaluate these two perspectives comprehensively. For example, based on image 19 of a drum performance, it is possible to evaluate the control of sound dynamics from information such as whether the movement pattern of player 2 (such as the swing range of the arms) is stable or not, and whether the movement pattern of stick 3 (such as the swing range of stick 3) is stable or not. In addition, it is possible to perform evaluation of this evaluation item based on information such as whether the movements of the performer 2 and the sticks 3 for a specific performance pattern (specific phrase) are close to a predetermined optimal pattern. As the extracted information, for example, skeletal information of the player 2, the movement of the sticks 3, etc. can be used. As the detected information, for example, sound production information, the movement of the performer, etc. can be used. As the auxiliary information, for example, correct answer information on the performance can be used. In addition, the information and parameters used to evaluate the control of sound dynamics are not limited and may be set arbitrarily.

[0055] (Tone control) It is possible to evaluate how well performer 2 controls the tone color. In Fig. 13, whether or not the intended tone color can be produced within the same instrument (drum) is listed as an evaluation item. For example, hit the center of the snare drum 6. Deliberately hit a part of the snare drum 6 that is off-center. Adjust the depth of the open rim shot. Adjust the position where the stick 3 hits the closed rim shot. Adjust the position where the hi-hat cymbal 10 is hit. Adjust the part of the stick 3 that hits the hi-hat cymbal 10. Adjust the amount of openness of the hi-hat cymbal 10. By making these kinds of performance adjustments, the performer 2 can control the tone. For example, based on image 19 of a drum performance, it is possible to evaluate the control of tone from information such as the position where the drum or cymbal is being struck, the part of the stick 3 that is hitting the drum, the movement pattern of the player 2 (such as the swing range of the arm), and the movement pattern of the stick 3 (such as the swing range of the stick 3). Furthermore, it is possible to perform evaluation of this evaluation item for a specific performance pattern (specific phrase) based on the movement of the player 2, the movement of the sticks 3, and the like. As the extracted information, for example, skeletal information of the player 2, the movement of the sticks 3, etc. can be used. As the detected information, for example, sound production information, the movement of the performer, etc. can be used. As the auxiliary information, for example, correct answer information on the performance can be used. In addition, the information and parameters used to evaluate the control of the tone color are not limited and may be set arbitrarily.

[0056] (Repeat performance stability) It is possible to evaluate how stable the performer 2 is when performing a repeated performance. A repeated performance is when the same performance is repeated, such as repeating parts of the same song or playing the same song multiple times. In Figure 13, whether there is little variation in the repeated performance is listed as an evaluation item. For example, based on image 19 of a drum performance, it is possible to evaluate the stability of a repeated performance from information such as whether the movement pattern of player 2 (such as the swing range of the arms) is stable or not for the repeated performance, and whether the movement pattern of stick 3 (such as the swing range of stick 3) is stable or not. As the extracted information, for example, skeletal information of the player 2, the movement of the sticks 3, etc. can be used. As the detected information, for example, sound production information, the movement of the performer, etc. can be used. As the auxiliary information, for example, correct answer information on the performance or past performance information can be used. In addition, the information and parameters used to evaluate the stability of repeated performances are not limited and may be set arbitrarily.

[0057] (Whether or not there is communication with other performance sounds) It is possible to evaluate how well performer 2 communicates with other performance sounds. Other performance sounds typically refer to the performance sounds of other musical instruments. This is not limited to this, but also includes performance sounds in which other performers are playing the same instrument (e.g., twin drums). In Figure 13, whether or not the musicians were able to communicate with the other parts besides the drummer is listed as an evaluation item. For example, there may be cases where intentional fluctuations, pauses, deviations, accelerations, decelerations, improvisational variations, etc. are produced between the drum performance and other sounds (other performers). It is possible to evaluate whether such various forms of communication are established between the drum performance and other sounds based on the image 19 of the drum performance. The extracted information can be the performer's facial expression, etc. Furthermore, if other performers or other instruments are included in the image 19, the facial expressions of the other performers and the performance status of the other instruments can be extracted as extracted information and used for evaluation. As the detected information, for example, information on the sound produced, information on other sounds being played, the movement of the performer, the movement of other performers, etc. can be used. As the auxiliary information, for example, information on correct performance or information on other performance sounds can be used. Other information, parameters, etc. used to evaluate whether or not there is communication with other played sounds are not limited and may be set arbitrarily.

[0058] FIG. 14 shows an example of evaluation items related to the movement of the stick 3 used in playing the drums. Evaluation items related to the movement of the stick 3 include, for example, rebound control, accuracy of stick playing technique, type of stick usable for playing, accuracy of performance, etc. Scores can be calculated for these evaluation items, making it possible to evaluate the proficiency of drumming with high precision.

[0059] (Rebound control) After the attack, it is possible to evaluate how well the rebound of Stick 3 is controlled. In Figure 14, how well the rebound is picked up is listed as an evaluation item. For example, based on the image 19 of a drum performance, it is possible to evaluate rebound control by obtaining the ratio of the speed of the stick 3 after the attack to the speed of the stick 3 before the attack. For example, when evaluating rebound control for a full stroke or upstroke, the greater the ratio of "stick 3 speed after the attack" to "stick 3 speed before the attack," the higher the score can be. For downstrokes and tap strokes, rebound control can also be evaluated based on the ratio of the speed of the stick 3 after the attack to the speed of the stick 3 before the attack. It is also possible to evaluate rebound control from the physical state of player 2 before and after the attack based on image 19 of the drum performance. As the extracted information, for example, the movement of the sticks 3, skeletal information of the player 2, etc. can be used. As the detected information, for example, sound production information can be used. For example, information on whether an appropriate sound is being produced according to the type of stroke can be used to evaluate rebound control. In addition, the movement of the player 2 and the physical condition of the player 2 (muscle relaxation / tension, etc.) can also be used as the detected information. As the auxiliary information, for example, correct answer information on the performance can be used. Other information, parameters, etc. used to evaluate rebound control are not limited and may be set arbitrarily.

[0060] (Accuracy of stick technique) It is possible to evaluate the accuracy of various stick playing techniques (e.g., single strokes, double strokes, paradiddles, ghost notes, rolls, etc.). In Figure 14, whether or not the intended stick playing technique is performed is listed as an evaluation item. For example, it is possible to evaluate the accuracy of stick playing from the pattern of the movement of the sticks 3 based on the image 19 of the drum performance. As the extracted information, for example, the movement of the sticks 3, skeletal information of the player 2, etc. can be used. As the detected information, for example, information on pronunciation, the movement of the player 2, and the physical state of the player 2 (muscle relaxation / tension, etc.) can be used. As the auxiliary information, for example, correct answer information on the performance can be used. In addition, the information, parameters, etc. used to evaluate the accuracy of stick playing are not limited and may be set arbitrarily.

[0061] (Types of sticks that can be used for playing) It is possible to evaluate whether or not a musical instrument can be played using not only regular sticks but also special sticks 3 such as brushes and broomsticks. In Fig. 14, whether or not a musical instrument can be played using each of the special sticks 3 is listed as an evaluation item. For example, based on the image 19 of a drum performance, it is possible to evaluate the types of sticks 3 used in the performance and the movements of the various sticks 3, and therefore the types of sticks that can be used in the performance. As the extracted information, for example, the movements of the various sticks 3, the skeletal information of the player 2, etc. can be used. As the detected information, for example, information on pronunciation, the movement of the player 2, and the physical state of the player 2 (muscle relaxation / tension, etc.) can be used. As the auxiliary information, for example, correct answer information on the performance can be used. In addition, the information, parameters, etc. used to evaluate the accuracy of stick playing are not limited and may be set arbitrarily.

[0062] (Performance accuracy) It is possible to evaluate the accuracy of a performance using the stick 3. For example, stick spinning can be used. In FIG. 14, whether the performance is refined or not is listed as an evaluation item. A performance using the stick 3 may be considered a wasteful movement in terms of musical performance, but it is possible to evaluate whether there is enough room for the wasteful movement and whether the performer can return to playing afterwards without any problems. For example, based on the image 19 of the drumming performance, it is possible to evaluate the accuracy of the performance from the movement patterns of the player 2 and the movement patterns of the sticks 3. As the extracted information, for example, the movement of the sticks 3, skeletal information of the player 2, etc. can be used. As the detected information, for example, information on pronunciation, the movement of the player 2, and the physical state of the player 2 (muscle relaxation / tension, etc.) can be used. As the auxiliary information, for example, correct answer information on the performance can be used. In addition, the information, parameters, etc. used to evaluate the accuracy of performance are not limited and may be set arbitrarily.

[0063] FIG. 15 shows an example of evaluation items related to the movements of the player 2. Evaluation items related to the movements of performer 2 include, for example, evaluation items related to the center of gravity, evaluation items related to how the body is used, evaluation items related to the stability of the performance, evaluation items related to the state of the body during performance, evaluation items related to sound production efficiency, or evaluation items related to communication with other performers.

[0064] Evaluation items relating to the center of gravity include, for example, the stability of the center of gravity. Evaluation items relating to body usage can include, for example, body usage according to pronunciation intervals, efficient body usage, and the like. An example of an evaluation item relating to the stability of performance is stability over a long period of time. Evaluation items regarding the physical condition during a performance can include, for example, whether or not the body is relaxed during the performance, the range of motion of each body part during the performance, and whether or not there is room for facial expression. As an evaluation item relating to pronunciation efficiency, for example, pronunciation efficiency itself can be cited. An example of an evaluation item regarding communication with other performers is whether or not there is eye contact with other performers. It is possible to calculate scores for these evaluation items, making it possible to evaluate drumming proficiency with high accuracy.

[0065] (Stability of center of gravity) It is possible to evaluate the stability of the center of gravity of player 2 while he or she is playing the drums. In Fig. 15, whether or not the center of gravity of the body is stable regardless of the intensity of the performance is listed as an evaluation item. Of course, whether or not the center of gravity of the body is stable may also be evaluated regardless of whether or not the performance is intense. For example, based on image 19 of a drum performance, it is possible to evaluate the stability of the center of gravity from information such as whether the center of gravity of player 2 is moving, and if so, whether the movement is regular or irregular. For example, if the movement of the center of gravity is regular, it is possible to evaluate the stability of the center of gravity as high. As the extracted information, for example, the center of gravity of the performer 2 can be used. As the detected information, for example, sound production information, the center of gravity of the player 2 (obtained from a center of gravity meter, for example), etc. can be used. As the auxiliary information, for example, correct answer information on the performance can be used. In addition, the information, parameters, etc. used to evaluate the stability of the center of gravity are not limited and may be set arbitrarily.

[0066] (How to use your body according to the pronunciation interval) It is possible to evaluate how the body is used in accordance with the sound interval. In Figure 15, whether or not the shoulder, upper arm, elbow, forearm, wrist, and fingers of player 2 are used appropriately in accordance with the sound interval for each of their right and left hands is listed as an evaluation item. In other words, it is evaluated whether or not the right and left hands are used appropriately in accordance with the sound interval. For example, based on image 19 of a drum performance, it is possible to evaluate how player 2 uses his or her body in accordance with the interval between sounds from information about each part of the body. For example, when playing the hi-hat cymbal 10 in short 16th notes, the shorter the interval between sounds, the more preferable the performance is to use the fingers. On the other hand, when playing the snare drum 6 in quarter notes, the longer the interval between sounds, the more preferable the performance is to use the shoulders. In this way, if it is possible to switch the parts of the body used in performance depending on the interval between sounds, it is possible to give a high evaluation (score) for the use of the body. Of course, it is possible to evaluate the use of the body in accordance with the tone interval, not limited to the right and left hands, but also for other parts such as the right and left feet, etc. For example, when playing the bass drum 5 with the right foot, it is possible to evaluate whether the groin, thighs, knees, calves, ankles, toes, etc. are used appropriately when playing fast (short tone interval) and when playing slowly (long tone interval). As the extracted information, for example, skeletal information of the player 2 (position of each part, movement (momentum), speed, acceleration, etc.) and the movement of the stick 3 can be used. The detected information may include, for example, sound production information, performance tempo, the movement of the performer (position of each part, movement (amount of momentum), speed, acceleration, etc.), the physical condition of the performer, etc. As the auxiliary information, for example, correct information on the performance (such as the performance tempo) can be used. In addition, the information, parameters, etc. used to evaluate how the body is used in accordance with the pronunciation interval are not limited and may be set arbitrarily.

[0067] (Efficient use of the body) It is possible to evaluate the efficiency of body use. In Figure 15, whether or not you can move between instruments with the least amount of movement is listed as an evaluation item. In other words, it evaluates whether or not the movement efficiency during performance is good. For example, based on image 19 of a drum performance, it is possible to evaluate the efficient use of the body from information on how the body is used for a specific performance pattern, such as rolling the toms. It is not necessarily the case that moving the shortest distance is sufficient, but a high score is awarded if the movement is made with the least amount of effort based on the body's mechanics. Furthermore, just as with keyboard fingering, efficient body movements are known for drumming. For example, an orthodox procedure is known, such as alternating between the right and left hands when playing the toms. The closer the technique is to such a procedure known as efficient body movements, the higher the rating will be. As the extracted information, for example, skeletal information of the player 2, the movement of the sticks 3, etc. can be used. As the detected information, for example, information on the pronunciation, the movement of the performer, the physical condition of the performer, etc. can be used. As the auxiliary information, for example, correct answer information on the performance can be used. In addition, the information, parameters, etc. used to evaluate efficient body use are not limited and may be set arbitrarily.

[0068] (Stability over long periods of playing) It is possible to evaluate the stability of drum performance over a long period of time. In Figure 15, whether or not the drum performance remains stable over a long period of time is listed as an evaluation item. For example, based on image 19 of a drum performance, it is possible to evaluate the stability of a long-term performance from the movements of the player 2 over time, the physical condition of the player 2, and the movements of the sticks 3. It is also possible to evaluate the stability of a long-term performance by comparing the movements of the player 2 and sticks 3 at a certain time with the movements of the player 2 and sticks 3 after a predetermined time has passed. As the extracted information, for example, skeletal information of the player 2, the movement of the sticks 3, etc. can be used. As the detected information, for example, information on pronunciation, performance tempo, movement of the performer, physical condition of the performer, etc. can be used. As the auxiliary information, for example, correct answer information on the performance can be used. In addition, the information and parameters used to evaluate the stability of a performance over a long period of time are not limited and may be set arbitrarily.

[0069] (Whether or not you feel weak during the performance) It is possible to evaluate whether or not the body is relaxed while playing the drums. In other words, it is possible to evaluate how relaxed the body is. In Figure 15, whether or not the body is relaxed regardless of the intensity of the performance is listed as an evaluation item. When playing drums, the drummer does not always relax his / her muscles, and there may be times when he / she plays with more force. In such cases, it is possible to evaluate whether or not the player is able to relax his / her muscles at the appropriate timing. For example, if the player is able to relax his / her muscles immediately after the attack or when not playing, a high score is given. For example, based on the image 19 of the drum performance, it is possible to evaluate whether or not the player 2 is relaxed during the performance from the movement of the player 2, the physical condition of the player 2, the movement of the sticks 3, etc. For example, by determining whether or not the muscles are stiff, it is possible to evaluate the presence or absence of relaxation and the degree of relaxation. As the extracted information, for example, skeletal information of the player 2, the movement of the sticks 3, etc. can be used. As the detected information, for example, information on pronunciation, the movement of the player 2, or the physical state of the player 2 (muscle relaxation / tension, etc.) can be used. As the auxiliary information, for example, correct answer information on the performance can be used. In addition, the information, parameters, etc. used to evaluate whether or not a relaxation has occurred during playing are not limited and may be set arbitrarily.

[0070] (Range of motion of each part of the body while playing) It is possible to evaluate the range of motion of each body part while playing the drums. In Figure 15, whether the range of motion of each body part is wide or whether it is widely used is listed as an evaluation item. It is known that the basic approach to drumming is to keep the armpits tight, since opening the armpits narrows the range of motion. For example, based on this approach, it is possible to evaluate a state in which the armpits are tight as a high evaluation state. Of course, depending on the drumming technique, there may be different ideas about the range of motion of each part of the body. For example, it may be thought that it is better to open the armpits. In any case, by setting appropriate evaluation criteria (for example, by appropriately controlling the score settings included in the training data), it is possible to carry out evaluations based on various ideas. This also applies to other evaluation items. For example, based on the image 19 of the drum performance, it is possible to evaluate the range of motion of each part of the body during the performance from the movements of the player 2, the state of the player's body, and the like. As the extracted information, for example, skeletal information of the player 2 can be used. As the detected information, for example, information on the pronunciation, the movement of the player 2, or the physical state of the player 2 can be used. As the auxiliary information, for example, correct answer information on the performance can be used. In addition, the information and parameters used to evaluate the range of motion of each part during performance are not limited and may be set arbitrarily.

[0071] (Phonetic efficiency) It is possible to evaluate the sound generation efficiency of drum performance. In Figure 15, whether a loud sound can be produced with minimal movement is listed as an evaluation item. For example, if a loud sound can be produced with minimal movement, it is considered to have good sound generation efficiency and is given a high rating. For example, based on the image 19 of the drum performance, the movement of the player 2, the physical condition of the player 2, etc. are acquired. Then, by detecting sound information as the detected information, it is possible to evaluate the sound generation efficiency. As the extracted information, for example, skeletal information of the player 2 can be used. As other detected information, for example, the movement of the player 2 or the state of the player's body can also be used. As the auxiliary information, for example, correct answer information on the performance can be used. In addition, the information and parameters used to evaluate the pronunciation efficiency are not limited and may be set arbitrarily.

[0072] (Whether or not there is eye contact with other performers) It is possible to evaluate whether or not a performer makes eye contact with other performers. In Figure 15, whether or not a performer makes eye contact with other performers while playing is listed as an evaluation item. Generally, a performer 2 who maintains eye contact with other members while performing is reassuring to watch and is often highly skilled. Also, a performer 2 who maintains eye contact is often well aware of the situation around them. From this perspective, a performer who maintains eye contact is given a high rating. For example, based on image 19 of a drum performance, it is possible to evaluate whether or not there is eye contact with other performers from the facial movements and directions of performer 2. If other performers are included in image 19, it is also possible to evaluate whether or not there is eye contact by using the facial movements and directions of the other performers. As the extracted information, for example, skeletal information of the performer 2, the performer's facial expression, etc. can be used. Furthermore, if other performers are included in the image 19, the facial expressions of the other performers can be extracted as extracted information and used for evaluation. As the detected information, for example, the movement of the performer, the movement of other performers, etc. can be used. As the auxiliary information, for example, information on correct performance or information on other performance sounds can be used. Other information, parameters, etc. used to evaluate whether or not eye contact has been made with other performers are not limited and may be set arbitrarily.

[0073] (Whether the facial expression is relaxed or not) It is possible to evaluate whether or not the facial expression of player 2 playing the drums is relaxed. In Fig. 15, whether or not the facial expression is relaxed is listed as an evaluation item. For example, if the face is smiling, the mouth is open, or there is no tension in the face (such as a frown), the facial expression is deemed to be relaxed and a high evaluation can be given. For example, based on the image 19 of the drum performance, it is possible to evaluate from the facial expression of the player 2 whether or not the facial expression is relaxed. As the extracted information, for example, the facial expression of the performer 2 can be used. As the detected information, for example, the movement of the performer can be used. As the auxiliary information, for example, correct answer information of the performance can be used. In addition, the information, parameters, etc. used to evaluate whether or not the facial expression is relaxed are not limited and may be set arbitrarily.

[0074] In addition, scores can be assigned to various evaluation items other than those shown in the tables of FIGS. For example, it is possible to evaluate the proficiency of a drummer based on the movement of the foot that steps on the hi-hat cymbal 10. When a player keeps rhythm by moving the foot that steps on the hi-hat cymbal 10 (without opening and closing the hi-hat cymbal 10), a high evaluation can be given if the movement to keep rhythm is accurate. Conversely, if the foot that steps on the hi-hat cymbal 10 moves irregularly due to inertia, a low evaluation can be given. Also, if the drummer can play the drums while keeping the foot that presses the hi-hat cymbal 10 still, a high score will be given. Furthermore, when the sticks 3 are used to count at the beginning of a song, the accuracy of the count may be included as an evaluation item. If the count is accurate, a high evaluation will be given.

[0075] [Output of evaluation results and support information] The information processing device 15 can output the score calculated as the evaluation result for each evaluation item. The information processing device 15 can also output support information for improving proficiency. By using the support information, player 2 can efficiently improve his / her proficiency in playing the drums. When the score and support information are output, an output unit 41 is configured as a functional block, as shown in FIG. The output unit 41 is configured by a CPU or the like executing a predetermined program, similar to the first acquisition unit 16, the evaluation unit 17, and the second acquisition unit 40. Of course, to realize the output unit 41, dedicated hardware such as an IC (integrated circuit) may be used.

[0076] 16 and 17 are schematic diagrams showing examples of output of evaluation results and support information. 16 and 17 show a case where a smartphone 34 is used as an embodiment of the information processing device according to the present invention. The smartphone 34 functions as a device that integrates the imaging device 14 and the information processing device 15 shown in Fig. 1. In other words, the smartphone 34 functions as a computer having an imaging function.

[0077] For example, the player 2 downloads an application (application program) for using the drum practice assistance system 1 to the smartphone 34. For example, the player 2 inputs information such as an ID and a password to create an account for using the drum practice support system 1. Of course, creating an account may not be necessary.

[0078] The performer 2 takes images of his / her drum performance using the camera mounted on the smartphone 34. For example, he / she places the smartphone in front of the drum set 4 and starts the video recording mode. Then, he / she plays the drum set 4 to take images of his / her drum performance. Alternatively, the smartphone 34 can be used to capture images of the drum practice using a practice pad or the like. This allows the images of the drum practice to be captured as images of the drum performance. Of course, another person can also be used to capture images of the player 2 playing the drums.

[0079] Player 2 launches an application for using drum practice support system 1 and inputs the captured image of the drum performance. There are no limitations on the GUI (Graphical User Interface) or method for inputting the image of the drum performance into the application, and any GUI or method may be used.

[0080] A drum image input by the player 2 is input by a first acquisition unit 16 shown in FIG. The evaluation unit 17 evaluates the proficiency of drumming by executing processes using machine learning as exemplified in Figures 3 to 8, or processes using a rule base as exemplified in Figures 9 to 11, etc. For example, a score is calculated for each evaluation item as exemplified in Figures 13 to 15. At this time, the second acquisition unit 40 may acquire the detection information and auxiliary information, which may be used for the evaluation of each evaluation item. The evaluation result and support information are output by the output unit 41. For example, the evaluation result and support information are output by image or sound.

[0081] In the example shown in FIG. 16, the score for the overall evaluation is displayed as the evaluation result on the touch panel 35 of the smartphone 34 to the right of the text "Overall Evaluation" ("B" and "85 points" in FIG. 16). Additionally, the overall evaluation of the evaluation items related to the performance sound is displayed to the right of the text "Performance sound" ("B" and "84 points" in Figure 16). If you want to see the detailed scores for each evaluation item related to the performance sound, you can select each item button 36a to switch to a screen that displays the detailed scores for each evaluation item. Additionally, the overall evaluation of the evaluation items related to the movement of the sticks 3 used in playing the drums is displayed to the right of the text "Stick" ("A" and "94 points" in FIG. 16). If you want to see the detailed scores for each evaluation item related to the movement of the sticks 3 used in playing the drums, you can select each item button 36b to switch to a screen that displays the detailed scores for each evaluation item. Additionally, the overall evaluation of the evaluation items related to the performer's movements is displayed to the right of the text for "body usage" ("C" and "71 points" in Figure 16). If you want to see the detailed scores for each evaluation item related to the performer's movements, you can select each item button 36c to switch to a screen that displays the detailed scores for each evaluation item. As an image (screen) displaying detailed scores for each evaluation item, for example, a tabular image such as those shown in Figures 13 to 15 is displayed. Any other arbitrary configuration may be adopted.

[0082] In the example shown in FIG. 16, evaluation comments regarding the overall evaluation are displayed as support information below the text "Overall Evaluation." For example, evaluation comments such as "It's no good at all. Please practice." or "Your use of the stick is good, so try to improve how you use your body!" are displayed. Of course, the content of the evaluation comments is not limited, and any comment can be displayed. The evaluation comments may also be read aloud.

[0083] 17, a virtual image 37 is displayed as support information. The virtual image 37 is also called an AR image and is superimposed on a real object. 17, an image 19 of a drummer playing, which shows the left hand of a player 2 and the stick 3 held by the left hand, is used as a real object, and a virtual image 37 of the stick is superimposed on it. For example, the virtual image 37 of the stick is displayed as support information for improving stick control, so as to reproduce an accurate upstroke movement. In this way, in order to teach the player 2 the correct movements of the sticks 3 (movements that will increase the score) and the correct movements of the body, the correct movements of the sticks 3 and the body may be displayed as virtual images 37. In other words, as the virtual images 37 superimposed on the real objects, virtual images 37 relating to the movements of the sticks 3 used in playing drums and virtual images 37 relating to the movements of the player 2 may be displayed. This allows the player 2 to intuitively understand the correct movements that he or she should aim for, and allows the player 2 to efficiently improve his or her proficiency. The example shown in FIG. 17 can also be considered as an AR representation in which a virtual image 37 of a stick is superimposed on the left hand and stick 3 of the player 2, which are real objects. Of course, virtual images may be displayed as support information without being superimposed on real objects. For example, virtual images of the sticks 3 and body moving correctly may be displayed without being superimposed on real objects. Even in this case, the player 2 can understand the correct movements of the sticks 3 and body by viewing the virtual images.

[0084] As an embodiment of the information processing device according to the present invention, it is assumed that AR glasses (HMD) that can be worn on the head of the player 2 are used. In this case, the evaluation results and support information can be displayed on the display unit of the AR glasses. For example, when the body, sticks 3, drum set 4, etc. of the performer 2 playing the drums are in view, the body, etc. of the performer 2 can be treated as real objects, and virtual images 37 of the body and sticks 3 can be superimposed as support information. For example, it is possible to take an image 19 of the drum performance, evaluate each evaluation item, and output the evaluation results and support information (evaluation comments and virtual image 37) in real time in response to the drum performance.

[0085] As an embodiment of the information processing device according to the present invention, it is assumed that VR glasses (HMD) that can be worn on the head of the performer 2 are used. In this case, the evaluation results and support information can be displayed on the display unit of the VR glasses. For example, a 3D model image of the body, sticks 3, drum set 4, etc. of a performer 2 playing the drums may be displayed as a virtual image (VR image). At this time, a model image of the performer 2 dressed in the same clothes as a favorite musician, or a model image of the favorite musician himself may be displayed. Furthermore, a situation such as performing in front of a packed audience at a famous concert venue or hall may be realized using VR representation. Furthermore, as support information, virtual images (VR images) that serve as models for teaching correct movements may be displayed in a predetermined VR space. For example, 3D model images of the sticks 3 or body that serve as models may be output as support information. Of course, these virtual images that serve as support information may be superimposed on the virtual images of the player 2 and the sticks 3. Even when using VR glasses, it is possible to take an image 19 of the drum performance, evaluate each evaluation item, and output the evaluation results and support information (evaluation comments and virtual images) in real time according to the drum performance.

[0086] The support information is not limited to comments on the level of proficiency as shown in FIG. 16 or a virtual image 37 superimposed on a real object as shown in FIG. 17, but various other support information may be output. For example, a history of past evaluations of proficiency may be displayed. By displaying the score history for each evaluation item, player 2 can check his or her own growth. In other words, visualization of growth is realized, which can increase motivation to practice. It is also possible to check areas where you have become weaker or your weaknesses, which will allow you to create an efficient practice plan.

[0087] As described above, in the drum practice support system 1 and information processing device 15 according to this embodiment, the drum performance proficiency level is evaluated based on the drum performance image 19. This makes it possible to efficiently improve the drum performance proficiency level. For example, using a high-performance imaging device 14 or other detection devices, it is possible to evaluate proficiency with a high degree of accuracy for various evaluation items. On the other hand, as described with reference to Figures 16 and 17, it is also possible to easily use this drum practice support system 1 with a single smartphone 34. This makes it possible to efficiently improve the proficiency of a wide range of performers 2, from professional drummers to amateur drummers (including beginners), according to their level. As a result, the number of people who want to play drums will increase, which will lead to the spread of drum performance and raise the overall performance level.

[0088] <Other embodiments> The present invention is not limited to the above-described embodiment, and various other embodiments can be realized.

[0089] One evaluation item for the movements of the performer 2 can be how the performer uses their body in response to the performance tempo. For example, one evaluation item can be whether or not the performer is able to use their shoulders, upper arms, elbows, forearms, wrists, and fingers appropriately in response to the BPM. For example, based on the image 19 of the drum performance, it is possible to evaluate how the player 2 uses their body in response to the performance tempo from information on the parts of the body that they primarily use for playing. It is also possible to obtain the BPM itself from the movements of the player 2 and the movements of the sticks 3 extracted from the image 19 of the drum performance.

[0090] The movement of each part of the performer 2 relative to the performance tempo may be evaluated. In other words, evaluation may be performed for each part of the performer 2. For example, the relationship between shoulder movement and playing tempo, the relationship between upper arm movement and playing tempo, the relationship between elbow movement and playing tempo, the relationship between forearm movement and playing tempo, the relationship between wrist movement and playing tempo, the relationship between finger movement and playing tempo, etc. may be evaluated individually. This allows for a detailed evaluation of how each part of the body is used in response to the playing tempo, making it possible to evaluate drumming proficiency with high accuracy.

[0091] For example, it is possible to evaluate the movement of each part in relation to the performance tempo by processing using machine learning as exemplified in FIGS. In this case, a learning model may be generated for each body part to evaluate the relationship between the performance tempo and the body part. For example, a first learning model may be constructed to evaluate the relationship between body part 1 (shoulder information) and the performance tempo, a second learning model may be constructed to evaluate the relationship between body part 2 (upper arm information) and the performance tempo, and a third learning model may be constructed to evaluate the relationship between body part 3 (elbow information) and the performance tempo. In other words, suppose the body is divided into n parts. In order to evaluate the relationship between the performance tempo and each part, n learning models may be constructed. Of course, it is also possible to evaluate the relationship between the performance tempo and the upper body, and the relationship between the performance tempo and the lower body, using separate learning models.

[0092] Also, support information regarding the movement of each part of the body of the performer 2 in relation to the performance tempo may be output. For example, information such as how to move each part in relation to a certain performance tempo may be output for each part. In other words, correct movements (movements that will increase the score) in relation to the performance tempo may be output for each part. Of course, the virtual image 37 for each body part may be displayed so as to be superimposed on the real object. For example, a virtual image 37 of a shoulder making correct movements may be superimposed on the actual shoulder of the player 2. Such display of the virtual image 37 may be executed for each body part.

[0093] Detailed evaluation of body usage according to the sound generation interval may be performed for each part of the body. For example, evaluation may be performed for each part of the shoulder, upper arm, elbow, forearm, wrist, and fingers for the right hand that strikes the hi-hat cymbal 10 finely. Similarly, evaluation may be performed for each part of the shoulder, upper arm, elbow, forearm, wrist, and fingers for the left hand that strikes the snare drum 6 loudly. A learning model may be constructed for each part to evaluate the relationship between each part and the sound interval, and support information regarding the movement of each part relative to the sound interval may be output.

[0094] Reliability information may be added to the evaluation result (score) for each evaluation item. For example, depending on the content included in drum performance image 19, there may be evaluation items that can be evaluated with very high accuracy, and conversely, evaluation items that can only be evaluated with low accuracy. For example, suppose that an image 19 of a drum performance is used that shows only the entire drumstick 3 from the shoulder to the fingertips with high accuracy, and that makes it very easy to see the attack. However, suppose that the face of the performer 2 is barely visible. In this case, it is possible to calculate a score with high accuracy for evaluation items relating to the movement of the sticks 3 used in playing the drums, as shown in Fig. 14. Therefore, a high degree of reliability is assigned to these evaluation results. On the other hand, the accuracy of score calculation is low for evaluation items such as whether or not there is eye contact with other performers and whether or not the facial expression is relaxed, as shown in Figure 15. Therefore, low reliability is assigned to these evaluation items. In this way, the reliability of the evaluation results may be assigned, which allows the player 2 to grasp useful evaluation results and efficiently improve their proficiency.

[0095] FIG. 18 is a block diagram showing an example of the hardware configuration of a computer 60 that can be used as the information processing device 15. The computer 60 includes a CPU 61, a ROM (Read Only Memory) 62, a RAM 63, an input / output interface 65, and a bus 64 that interconnects these components. The input / output interface 65 is connected to a display unit 66, an input unit 67, a storage unit 68, a communication unit 69, a drive unit 70, and the like. The display unit 66 is a display device that uses, for example, a liquid crystal display, an electroluminescent display, etc. The input unit 67 is, for example, a keyboard, a pointing device, a touch panel, or other operating device. When the input unit 67 includes a touch panel, the touch panel can be integrated with the display unit 66. The storage unit 68 is a non-volatile storage device such as a HDD, flash memory, or other solid-state memory. The drive unit 70 is a device capable of driving a removable storage medium 71 such as an optical storage medium or magnetic recording tape. The communication unit 69 is a modem, router, or other communication device that can be connected to a LAN, WAN, or the like and that communicates with other devices. The communication unit 69 may communicate either wired or wirelessly. The communication unit 69 is often used separately from the computer 60. Information processing by computer 60 having the above hardware configuration is realized by software stored in storage unit 68, ROM 62, etc., working in cooperation with the hardware resources of computer 60. Specifically, the information processing method according to the present invention is realized by loading a program constituting the software stored in ROM 62, etc., into RAM 63 and executing it. The program is installed in the computer 60 via, for example, the recording medium 71. Alternatively, the program may be installed in the computer 60 via a global network or the like. Any other computer-readable non-transitory storage medium may be used.

[0096] The information processing method (drum practice support method, proficiency assessment method) and program of the present invention may be executed by multiple computers connected to each other so as to be able to communicate via a network or the like, thereby constructing an information processing device of the present invention. That is, the information processing method and program according to the present invention can be executed not only in a computer system configured by a single computer, but also in a computer system in which a plurality of computers operate in conjunction with each other. In this disclosure, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems. The execution of the information processing method and program according to the present invention by a computer system includes both cases where the acquisition of an image of a drum performance, the evaluation of drum performance proficiency, the extraction of extracted information, the acquisition of detected information, the acquisition of auxiliary information, the output of scores and support information, etc. are performed by a single computer, and cases where each process is performed by a different computer. Furthermore, the execution of each process by a specific computer includes having another computer execute part or all of the process and obtaining the results. That is, the information processing method and program according to the present invention can also be applied to a cloud computing configuration in which a single function is shared and processed jointly by a plurality of devices via a network.

[0097] The drum performance support system, information processing device, smartphone, GUI configuration for outputting evaluation results and support information, and processing flows for evaluating drum performance proficiency, etc., described with reference to the drawings are merely exemplary embodiments and may be modified as desired without departing from the spirit of the present invention. In other words, any other configurations, algorithms, etc. may be employed to implement the present invention.

[0098] In this disclosure, to facilitate understanding of the explanation, words such as "approximately," "almost," and "roughly" may be used as appropriate. However, there is no clear difference between using and not using words such as "approximately," "almost," and "roughly." That is, in the present disclosure, concepts that define shape, size, positional relationship, state, etc., such as "center," "central," "uniform," "equal," "same," "orthogonal," "parallel," "symmetrical," "extended," "axial direction," "cylindrical," "cylindrical," "ring-shaped," and "annular," are concepts that include "substantially center," "substantially central," "substantially uniform," "substantially equal," "substantially the same," "substantially orthogonal," "substantially parallel," "substantially symmetrical," "substantially extended," "substantially axial direction," "substantially cylindrical," "substantially cylindrical," "substantially ring-shaped," "substantially annular," and the like. For example, this also includes states that fall within a specified range (for example, a range of ±10%) based on criteria such as "perfectly centered," "perfectly central," "perfectly uniform," "perfectly equal," "perfectly the same," "perfectly perpendicular," "perfectly parallel," "perfectly symmetrical," "perfectly extended," "perfectly axial," "perfectly cylindrical," "perfectly cylindrical," "perfectly ring-shaped," and "perfectly annular." Therefore, even if the words "roughly," "almost," "approximately," etc. are not added, the concept expressed by adding "roughly," "almost," "approximately," etc. may be included. Conversely, a state expressed by adding "roughly," "almost," "approximately," etc. does not necessarily exclude a complete state.

[0099] In this disclosure, expressions using "more than," such as "greater than A" and "smaller than A," are expressions that comprehensively include both concepts that include equivalent to A and concepts that do not include equivalent to A. For example, "greater than A" is not limited to cases that do not include equivalent to A, but also includes "A or greater." Furthermore, "smaller than A" is not limited to "less than A," but also includes "A or less." When implementing the present technology, specific settings and the like may be appropriately adopted from the concepts included in "greater than A" and "smaller than A" so as to achieve the effects described above.

[0100] It is also possible to combine at least two of the features of the present invention described above. That is, the various features described in each embodiment may be combined in any way without distinguishing between the embodiments. Furthermore, the various effects described above are merely examples and are not limiting, and other effects may also be achieved. [Explanation of symbols]

[0101] 1. Drum practice support system 2…Performer 3...Stick 4...Drum set 14...imaging device 15...Information processing device 19...Image of drumming 20, 24, 28…Learning Model 34...Smartphone 37...Virtual image 60...Computer

Claims

1. a first acquisition unit that acquires an image of a drum performance by a performer; an evaluation unit that evaluates the level of proficiency in playing the drums based on the image acquired by the first acquisition unit; Equipped with the evaluation unit calculates a score indicating the proficiency level for each of one or more evaluation items related to the drum performance based on the extracted information extracted from the image; The extracted information includes facial expressions of the performer. Information processing device.

2. 2. The information processing device according to claim 1, The one or more evaluation items are evaluation items related to communication with other performers. Information processing device.

3. 3. The information processing device according to claim 1, The one or more evaluation items are evaluation items relating to whether or not the facial expression is relaxed. Information processing device.

4. 4. The information processing device according to claim 1, The extracted information includes at least one of one or more feature points related to the drum performance, skeletal information of the player, center of gravity of the player, and the body condition of the player. Information processing device.

5. 5. The information processing device according to claim 4, The one or more evaluation items include at least one of an evaluation item related to the sound of the drum, an evaluation item related to the movement of the drum sticks used in playing the drums, or an evaluation item related to the movement of the player. Information processing device.

6. 6. The information processing device according to claim 5, The evaluation items regarding the performance sounds include at least one of control of sound timing, control of sound dynamics, control of timbre, stability of repeated performance, and presence or absence of communication with other performance sounds. Information processing device.

7. 6. The information processing device according to claim 5, The evaluation items regarding the stick movement include at least one of rebound control, accuracy of stick playing technique, types of sticks that can be used for playing, and accuracy of performance. Information processing device.

8. 6. The information processing device according to claim 5, The evaluation items related to the player's movements include at least one of an evaluation item related to the center of gravity, an evaluation item related to the use of the body, an evaluation item related to the stability of the performance, an evaluation item related to the state of the body during the performance, an evaluation item related to the efficiency of sound production, and an evaluation item related to communication with other players. Information processing device.

9. 9. The information processing device according to claim 1, The evaluation unit evaluates the movement of each part of the performer relative to the performance tempo. Information processing device.

10. 10. The information processing device according to claim 9, The evaluation unit evaluates the movement of each part of the performer by processing using machine learning. Information processing device.

11. 11. The information processing device according to claim 9, The support information includes support information regarding the movement of each part of the performer relative to the performance tempo. Information processing device.

12. The information processing device according to any one of claims 1 to 11, further comprising: a second acquisition unit that acquires at least one of detection information detected in response to the drum performance and auxiliary information related to the drum performance; The evaluation unit calculates the evaluation using at least one of the detection information and the auxiliary information acquired by the second acquisition unit. Information processing device.

13. 13. The information processing device according to claim 12, The detected information includes at least one of sound information, performance time, performance tempo, sound interval, movement of the player, center of gravity of the player, or body condition of the player. Information processing device.

14. 14. The information processing device according to claim 12, The auxiliary information includes at least one of correct performance information, past performance information, and information on other performance sounds. Information processing device.

15. 15. The information processing device according to claim 1, further comprising: An output unit is provided that outputs support information for improving the level of proficiency. Information processing device.

16. 16. The information processing device according to claim 15, The support information includes at least one of a virtual image, a comment on the proficiency level, or a history of the proficiency level. Information processing device.

17. 17. The information processing device according to claim 16, The virtual image is superimposed on the real object. Information processing device.

18. 18. The information processing device according to claim 17, The virtual image is a virtual image relating to at least one of the movement of a stick used in playing the drums and the movement of a player. Information processing device.

19. 19. An information processing device according to claim 1, The evaluation unit evaluates the proficiency level at a portion where the rhythm changes in the composition of the song. Information processing device.

20. An information processing method executed by a computer system, comprising: Acquire an image of a performer playing a drum; evaluating the level of proficiency in playing the drums based on the acquired images; the process of evaluating the proficiency level includes calculating a score indicating the proficiency level for each of one or more evaluation items related to the drum performance based on extracted information extracted from the image; The extracted information includes facial expressions of the performer. Information processing methods.

21. A program for causing a computer system to execute an information processing method, The information processing method includes: Acquire an image of a performer playing a drum; evaluating the level of proficiency in playing the drums based on the acquired images; the process of evaluating the proficiency level includes calculating a score indicating the proficiency level for each of one or more evaluation items related to the drum performance based on extracted information extracted from the image; The extracted information includes facial expressions of the performer. program.

Images