Information processing methods, information processing systems, and programs

Abstract

To provide a technique capable of specifying an image of a player necessary for teaching a performance using a musical instrument.SOLUTION: An information processing method executed by a computer includes: determining a point of interest in a body of a player who plays a musical instrument based on information on a person who confirms a performance of the player; and acquiring image information representing an image of the determined point of interest.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present disclosure relates to an information processing method and an information processing system.

Background Art

[0002] Patent Document 1 discloses a performance evaluation device that automatically evaluates performances.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When teaching a performance using an instrument is performed using an image, it is important to identify an image of the performer necessary for the teaching. An object of the present disclosure is to provide a technique capable of identifying an image of a performer necessary for teaching.

Means for Solving the Problems

[0005] An information processing method according to an aspect of the present disclosure is an information processing method executed by a computer, which determines a point of attention on the body of a performer who plays the instrument indicated by the instrument information based on the instrument information indicating the instrument, and acquires image information representing an image of the determined point of attention.

[0006]

[0007] Another information processing method according to an aspect of the present disclosure is an information processing method executed by a computer, which determines a point of attention on the body of a performer who plays the instrument based on sound information indicating sound output from the instrument, and acquires image information indicating an image of the determined point of attention.An information processing system according to yet another aspect of this disclosure includes a determination unit that determines a point of interest from the body of a performer playing the instrument indicated by the instrument information, based on instrument information indicating the instrument, and an acquisition unit that acquires image information representing an image of the point of interest determined by the determination unit.

[0008] An information processing system according to yet another aspect of this disclosure includes a determination unit that determines a point of interest from the body of a performer playing a musical instrument based on sound information indicating the sound output from the musical instrument, and an acquisition unit that acquires image information representing an image of the point of interest determined by the determination unit. [Brief explanation of the drawing]

[0009] [Figure 1] This is a diagram showing an example of information provision system 1. [Figure 2] This figure shows an example of the student training system 100. [Figure 3] This figure shows an example of a correspondence table Ta. [Figure 4] This is a diagram illustrating the operation of the student training system 100. [Figure 5] This is a diagram showing student image G3. [Figure 6] This is a diagram illustrating the operation of the student training system 100. [Figure 7] This figure shows an example of the correspondence table Ta1. [Figure 8] This is a diagram showing the student training system 101. [Figure 9] This is a diagram illustrating how to crop an image to show a part of a performer's body. [Figure 10] This is a diagram showing the student training system 102. [Figure 11] This figure shows an example of tablature. [Figure 12] This is a diagram showing an example of a guitar chord chart. [Figure 13] This is a diagram showing an example of drum notation. [Figure 14] This figure shows an example of a piano duet score. [Figure 15] This is a diagram showing an example of musical notes indicating the simultaneous pronunciation of multiple sounds. [Figure 16] This is a diagram showing an example of a schedule indicated by schedule information. [Figure 17] This is a diagram showing another example of a schedule indicated by schedule information. [Figure 18] This is a diagram showing the student teaching system 103. [Figure 19] This is a diagram showing the student teaching system 104. [Figure 20] This is a diagram showing an example of a user interface. [Figure 21] This is a diagram showing the student teaching system 105. [Figure 22] This is a diagram showing an example of the learning processing unit 191. [Figure 23] This is a diagram showing an example of learning processing. [Figure 24] This is a diagram showing another example of the processing device 180.

Modes for Carrying Out the Invention

[0010] A: First Embodiment A1: Information Providing System 1 FIG. 1 is a diagram showing an example of the information providing system 1 of the present disclosure. The information providing system 1 is an example of an information processing system. The information providing system 1 includes a student teaching system 100 and a teacher guidance system 200. The student teaching system 100 and the teacher guidance system 200 can communicate with each other via a network NW. The configuration of the teacher guidance system 200 is the same as that of the student teaching system 100.

[0011] The student teaching system 100 is used by a student 100B who learns to play a music piece using a musical instrument 100A. The student teaching system 100 is arranged in a student room provided in a music classroom. The student teaching system 100 may be arranged at a location different from the student room provided in the music classroom, for example, at the home of the student 100B.

[0012] Instrument 100A is either a piano or a flute. The piano and flute are examples of instrument types and examples of instruments, respectively. Hereafter, the phrase "instrument type" can be replaced with the phrase "instrument." Student 100B is an example of a performer. The location where student 100B plays instrument 100A is predetermined in the room where the student training system 100 is located. Therefore, student 100B during performance, student 100B immediately before performance, and student 100B immediately after performance can be imaged by a fixed camera.

[0013] The teacher instruction system 200 is used by teacher 200B, who instructs students on playing musical pieces using instrument 200A. The type of instrument 200A is the same as the type of instrument 100A. For example, if instrument 100A is a piano, then instrument 200A is also a piano. The teacher instruction system 200 is placed in a teacher's room in the music classroom. The teacher instruction system 200 may also be placed in a different location from the teacher's room in the music classroom, for example, at teacher 200B's home.

[0014] Teacher 200B is an example of a performer. The location where Teacher 200B plays instrument 200A is predetermined in the room where the teacher instruction system 200 is located. Therefore, Teacher 200B during performance, immediately before performance, and immediately after performance can be captured by a fixed camera.

[0015] The student instruction system 100 transmits student performance information a to the teacher instruction system 200. Student performance information a indicates the situation in which student 100B is playing instrument 100A. Student performance information a includes student image information a1 and student sound information a2.

[0016] Student image information a1 shows an image (hereinafter referred to as "student image") representing student 100B playing instrument 100A. Student sound information a2 shows the sound (hereinafter referred to as "student performance sound") output from instrument 100A when student 100B is playing instrument 100A.

[0017] The teacher instruction system 200 receives student performance information a from the student instruction system 100. The teacher instruction system 200 displays a student image based on the student image information a1 contained in the student performance information a. The teacher instruction system 200 outputs a student performance sound based on the student sound information a2 contained in the student performance information a.

[0018] The teacher instruction system 200 transmits teacher performance information b to the student instruction system 100. Teacher performance information b indicates the situation in which teacher 200B is playing instrument 200A. Teacher performance information b includes teacher image information b1 and teacher sound information b2.

[0019] Teacher image information b1 shows an image (hereinafter referred to as "teacher image") representing the situation in which teacher 200B is playing instrument 200A. Teacher sound information b2 shows the sound of the music (hereinafter referred to as "teacher performance sound") output from instrument 200A in the situation in which teacher 200B is playing instrument 200A.

[0020] The student instruction system 100 receives teacher performance information b from the teacher instruction system 200. The student instruction system 100 displays a teacher image based on the teacher image information b1 contained in the teacher performance information b. The student instruction system 100 outputs a teacher performance sound based on the teacher sound information b2 contained in the teacher performance information b.

[0021] A2: Student Training System 100 Figure 2 shows an example of the student training system 100. The student training system 100 includes cameras 111-115, a microphone 120, a display unit 130, a speaker 140, an operation unit 150, a communication unit 160, a storage device 170, and a processing device 180.

[0022] Each of the cameras 111 to 115 includes an image sensor that converts light into electrical signals. The image sensor is, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

[0023] Camera 111 generates student finger information a11 by imaging each finger of student 100B's hand operating the instrument 100A. Student finger information a11 represents each finger of student 100B's hand operating the instrument 100A, as well as the instrument 100A itself, in image form.

[0024] Camera 112 generates student foot information a12 by imaging both feet of student 100B who is operating the instrument 100A. Student foot information a12 represents the feet of student 100B who is operating the instrument 100A, as well as the instrument 100A, in image form.

[0025] Camera 113 generates student whole-body information a13 by imaging the entire body of student 100B who is operating the instrument 100A. Student whole-body information a13 represents the entire body of student 100B operating the instrument 100A, as well as the instrument 100A itself, in image form.

[0026] Camera 114 generates student mouth information a14 by capturing an image of the mouth of student 100B operating the instrument 100A. Student mouth information a14 represents the mouth of student 100B operating the instrument 100A and the instrument 100A itself as images.

[0027] Camera 115 generates student upper body information a15 by capturing images of the upper body of student 100B operating the instrument 100A. Student upper body information a15 represents the upper body of student 100B operating the instrument 100A and the instrument 100A itself in images.

[0028] At least one of the following is included in the student image information a1: student finger information a11, student foot information a12, student whole body information a13, student mouth information a14, and student upper body information a15. The orientation and posture of cameras 111-115 are adjustable. Each of cameras 111-115 is also referred to as an imaging unit.

[0029] Microphone 120 captures the sound of the students playing. Based on the sound of the students playing, microphone 120 generates student sound information a2. Microphone 120 is also referred to as the sound pickup unit.

[0030] The display unit 130 is a liquid crystal display. The display unit 130 is not limited to a liquid crystal display; for example, it may be an OLED (Organic Light Emitting diode) display. The display unit 130 may also be a touch panel. The display unit 130 displays various types of information. For example, the display unit 130 may display a teacher image based on teacher image information b1. The display unit 130 may also display a student image based on student image information a1.

[0031] Speaker 140 outputs various sounds. For example, speaker 140 outputs teacher performance sounds based on teacher sound information b2. Speaker 140 may also output student performance sounds based on student sound information a2.

[0032] The control unit 150 is a touch panel. The control unit 150 is not limited to a touch panel; for example, it may be various operation buttons. The control unit 150 receives various information from a user, such as student 100B. The control unit 150 receives, for example, student instrument information c1 from the user. Student instrument information c1 indicates the type of instrument 100A. Student instrument information c1 is an example of instrument information indicating the type of instrument.

[0033] The communication unit 160 communicates with the teacher guidance system 200 via a network NW, either by wired or wireless connection. The communication unit 160 may also communicate with the teacher guidance system 200 via a wired or wireless connection without using the network NW. The communication unit 160 transmits student performance information a to the teacher guidance system 200. The communication unit 160 receives teacher performance information b from the teacher guidance system 200.

[0034] The storage device 170 is a computer-readable recording medium (for example, a computer-readable non-transitory recording medium). The storage device 170 includes one or more memories. The storage device 170 includes, for example, non-volatile memory and volatile memory. Non-volatile memory is, for example, ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory), and EEPROM (Electrically Erasable Programmable Read Only Memory). Volatile memory is, for example, RAM (Random Access Memory).

[0035] The memory device 170 stores a processing program, an arithmetic program, and various data. The processing program defines the operation of the student training system 100. The arithmetic program defines the calculation that identifies the output Y1 from the input X1.

[0036] The storage device 170 may store processing programs and arithmetic programs read from a storage device in a server (not shown). In this case, the storage device in the server is an example of a computer-readable recording medium (e.g., a computer-readable non-transitory recording medium). Various data include several variables K1, which will be described later.

[0037] The processing unit 180 includes one or more CPUs (Central Processing Units). One or more CPUs are an example of one or more processors. Each of the processing unit, processor, and CPU is an example of a computer. Some or all of the functions of the processing unit 180 may be implemented by circuits such as DSPs (Digital Signal Processors), ASICs (Application Specific Integrated Circuits), PLDs (Programmable Logic Devices), and FPGAs (Field Programmable Gate Arrays).

[0038] The processing unit 180 reads the processing program and the arithmetic program from the storage device 170. By executing the processing program, the processing unit 180 functions as a specific unit 181, a determination unit 183, an acquisition unit 184, a transmission unit 185, and an output control unit 186. By executing the arithmetic program and using multiple variables K1, the processing unit 180 functions as a trained model 182. The processing unit 180 is an example of an information processing device.

[0039] The identification unit 181 identifies student instrument information c2 using student sound information a2. Student instrument information c2 indicates the type of instrument 100A. Student instrument information c2 is an example of instrument information indicating the type of instrument. Instrument information indicating the type of instrument (e.g., piano) is an example of instrument information indicating an instrument (e.g., piano). Student sound information a2 is an example of related information related to the type of instrument. Related information related to the type of instrument (e.g., piano) is an example of related information about an instrument (e.g., piano). If student sound information a2 indicates the sound of a piano, the identification unit 181 identifies student instrument information c2 indicating piano as the type of instrument 100A. The identification unit 181 identifies student instrument information c2, for example, by using a trained model 182.

[0040] The trained model 182 is composed of a neural network. For example, the trained model 182 is composed of a deep neural network (DNN). The trained model 182 may also be composed of a convolutional neural network (CNN). Both the deep neural network and the convolutional neural network are examples of neural networks. The trained model 182 may be composed of a combination of multiple types of neural networks. The trained model 182 may have additional elements such as Self-Attention. The trained model 182 may not be composed of a neural network, but rather of a Hidden Markov Model (HMM) or a Support Vector Machine (SVM).

[0041] The trained model 182 has learned the relationship between first information related to the type of instrument and second information indicating the type of instrument to which the first information is related. The first information is an example of training-related information about an instrument. The second information is an example of training-instrument information indicating the instrument identified from the training-related information. The trained model 182 uses output sound information indicating the sound output by the instrument as the first information. The trained model 182 uses information indicating the type of instrument that outputs the sound indicated by the output sound information as the second information. The trained model 182 is an example of the first trained model.

[0042] The multiple variables K1 used to realize the trained model 182 are identified by machine learning using multiple training data T1. The training data T1 includes a combination of training input data and training output data. The training data T1 includes first information as training input data. The training data T1 includes second information as training output data. An example of training data T1 is a combination of output sound information (first information) indicating the sound output by an instrument, and information (second information) indicating the type of instrument that produces the sound indicated by the output sound information.

[0043] The trained model 182 generates an output Y1 corresponding to the input X1. The trained model 182 uses "related information related to the type of instrument (e.g., student sound information a2)" as the input X1, and "information indicating the type of instrument that produces the sound indicated by the related information" as the output Y1.

[0044] Note that multiple training data T1 may have only training input data (first information) and no training output data (second information). In this case, multiple variables K1 are identified by machine learning so that the multiple training data T1 can be divided into multiple clusters based on the similarity of the multiple training data T1. Then, in the trained model 182, second information appropriate to each cluster is associated by a human. The trained model 182 identifies a cluster according to the input X1 and generates second information corresponding to the identified cluster as output Y1.

[0045] The determination unit 183 determines a point of focus from the body of a performer (e.g., student 100B) who uses the type of instrument indicated by the instrument information (student instrument information c1 or c2), based on the instrument information. A performer who uses the type of instrument indicated by the instrument information is an example of a performer who plays the instrument indicated by the instrument information. A point of focus is a part of the body that is noticed by the teacher for the type of instrument indicated by the instrument information. The determination unit 183 determines the point of focus by referring to a correspondence table Ta that shows the correspondence between the type of instrument and the body part (point of focus). A point of focus is at least one of the following, for example, each finger of student 100B, both feet of student 100B, the whole body of student 100B, the mouth of student 100B, and the upper body of student 100B. The correspondence table Ta is stored in the storage device 170.

[0046] The acquisition unit 184 acquires various types of information. For example, the acquisition unit 184 acquires image information representing the image of the area of ​​interest determined by the decision unit 183. The acquisition unit 184 acquires information representing the image of the area of ​​interest determined by the decision unit 183 from among student finger information a11, student foot information a12, student whole body information a13, student mouth information a14, and student upper body information a15 as target image information. Target image information is an example of image information. The acquisition unit 184 generates student image information a1 using the target image information. For example, the acquisition unit 184 generates student image information a1 that includes the target image information.

[0047] The transmitting unit 185 transmits the student image information a1 generated by the acquisition unit 184 to the teacher guidance system 200 via the communication unit 160. The teacher guidance system 200 is an example of a destination. The destination is an example of an external device.

[0048] The output control unit 186 controls the display unit 130 and the speaker 140. For example, the output control unit 186 displays a teacher image on the display unit 130 based on teacher image information b1. In this case, first, the acquisition unit 184 acquires the teacher image information b1 from the communication unit 160. The acquisition unit 184 provides the teacher image information b1 to the output control unit 186. The output control unit 186 uses the teacher image information b1 to display the teacher image on the display unit 130.

[0049] The output control unit 186 may display a student image on the display unit 130 based on the student image information a1. In this case, the acquisition unit 184 provides the student image information a1 to the output control unit 186. The output control unit 186 uses the student image information a1 to display a student image on the display unit 130. In this case, even if teacher 200B is absent, student 100B can independently learn to play the instrument 100A by looking at the student image (image of the area of ​​interest) indicated by the student image information a1. Furthermore, if at least the student instruction system 100 exists but the teacher instruction system 200 does not, student 100B can independently learn to play the instrument 100A by looking at the student image (image of the area of ​​interest) indicated by the student image information a1.

[0050] The output control unit 186 may display the teacher image and student image side by side on the display unit 130 based on the teacher image information b1 and student image information a1. In this case, the acquisition unit 184 acquires the teacher image information b1 and student image information a1 as described above. The acquisition unit 184 provides the teacher image information b1 and student image information a1 to the output control unit 186. The output control unit 186 displays the teacher image and student image side by side on the display unit 130 based on the teacher image information b1 and student image information a1.

[0051] The output control unit 186 outputs the teacher's performance sound to the speaker 140 based on the teacher's sound information b2. In this case, first, the acquisition unit 184 acquires the teacher's sound information b2 from the communication unit 160. The acquisition unit 184 provides the teacher's sound information b2 to the output control unit 186. The output control unit 186 uses the teacher's sound information b2 to output the teacher's performance sound to the speaker 140.

[0052] The output control unit 186 may output the student performance sound to the speaker 140 based on the student sound information a2. In this case, first, the acquisition unit 184 acquires the student sound information a2 from the microphone 120. The acquisition unit 184 provides the student sound information a2 to the output control unit 186. The output control unit 186 uses the student sound information a2 to output the student performance sound to the speaker 140.

[0053] The output control unit 186 may alternately output teacher performance sounds and student performance sounds to the speaker 140 based on teacher sound information b2 and student sound information a2. In this case, the acquisition unit 184 acquires teacher sound information b2 and student sound information a2 as described above. The acquisition unit 184 provides teacher sound information b2 and student sound information a2 to the output control unit 186. The output control unit 186 alternately outputs teacher performance sounds and student performance sounds to the speaker 140 based on teacher sound information b2 and student sound information a2.

[0054] A3: Teacher Guidance System 200 The teacher instruction system 200 differs from the student instruction system 100 in that it is used by teacher 200B rather than student 100B. The configuration of the teacher instruction system 200 is the same as that of the student instruction system 100, as described above.

[0055] The main explanation of the structure of the teacher instruction system 200 is achieved by making the following substitutions in the explanation of the student instruction system 100 described above: Replace "instrument 100A" with "instrument 200A". Replace "student 100B" with "teacher 200B". Replace "student performance information a" with "teacher performance information b". Replace "student image information a1" with "teacher image information b1". Replace "student finger information a11" with "teacher finger information b11". Replace "student foot information a12" with "teacher foot information b12". Replace "student whole body information a13" with "teacher whole body information b13". Replace "student mouth information a14" with "teacher mouth information b14". Replace "student upper body information a15" with "teacher upper body information b15". Replace "student sound information a2" with "teacher sound information b2". Replace "student instrument information c1, c2" with "teacher instrument information d1, d2". Replace "teacher performance information b" with "student performance information a". Replace "teacher image information b1" with "student image information a1". Replace "teacher sound information b2" with "student sound information a2". For this reason, a detailed explanation of the structure of the teacher instruction system 200 will be omitted.

[0056] A4: Compatible Table Ta Figure 3 shows an example of a correspondence table Ta. The correspondence table Ta shows the relationship between the type of instrument and the body part (point of focus). The "Type of Instrument" column in the correspondence table Ta shows the type of instrument that is the subject of instruction. The correspondence table Ta shows "piano" and "flute" as the types of instruments. The "Body Part (Point of Focus)" column in the correspondence table Ta shows the body part of the performer that is required as an image in instruction for the instrument shown in the "Type of Instrument" column.

[0057] In piano lessons, students face the piano in a posture they prefer, press the keys with their fingers, and operate the damper pedal with their feet. The teacher focuses on each of the student's fingers, both of their feet, and their entire body (e.g., posture) to guide the student. For example, the teacher focuses on each of the student's fingers to guide the movement of each finger in a musical passage. The teacher focuses on the student's feet to guide the operation of the damper pedal. The teacher focuses on the positional relationship between each of the student's fingers and the keys to guide correct keystrokes. The teacher focuses on the student's entire body to guide the student's posture while playing. The teacher guides the student by showing them at least one of the following: the teacher's fingers, both of their feet, and the teacher's entire body (posture, etc.). Therefore, in the correspondence table Ta, the instrument type "piano" is associated with the body parts "each of the hands, both of the feet, and the entire body."

[0058] In flute lessons, the student positions the flute near their upper body, blows air into the flute through their mouth, and operates the keys with their fingers. The teacher focuses on the student's mouth and upper body (e.g., posture, angle between the student and the flute, and finger placement) to guide the student. For example, the teacher focuses on the student's mouth to guide the shape of the lips while playing. The teacher focuses on the student's upper body to guide the student's positional relationship with the flute. The teacher guides the student by showing them either the teacher's mouth or at least one of the teacher's upper body. Therefore, in correspondence table Ta, the instrument type "flute" is associated with the body part "mouth and upper body".

[0059] A5: Operation of Student Training System 100 Figure 4 is a diagram illustrating the operation of the student instruction system 100 in transmitting student performance information a. The storage device 170 stores imaging target information indicating the respective imaging targets of cameras 111 to 115.

[0060] Student 100B plays instrument 100A so that the student instruction system 100 can identify the type of instrument 100A. In step S101, microphone 120 generates student sound information a2 based on the sound output from instrument 100A.

[0061] Next, in step S102, the identification unit 181 uses the student sound information a2 to identify student instrument information c2, which indicates the type of instrument 100A.

[0062] In step S102, the identification unit 181 first inputs student sound information a2 into the trained model 182. Subsequently, the identification unit 181 identifies the information that the trained model 182 outputs in response to the input of student sound information a2 as student instrument information c2.

[0063] Next, in step S103, the determination unit 183 determines the area of ​​focus from the body of the performer student 100B based on the student instrument information c2.

[0064] In step S103, the determination unit 183 determines, in the correspondence table Ta, the body parts corresponding to the type of instrument indicated by the student instrument information c2 as points of interest. For example, if the student instrument information c2 indicates a piano, the determination unit 183 determines each finger of student 100B's hand, both feet of student 100B, and each of student 100B's entire body as points of interest for student 100B.

[0065] Furthermore, if the operation unit 150 receives student instrument information c1 indicating the type of instrument 100A from a user such as student 100B, the determination unit 183 may determine the point of interest on student 100B's body based on the student instrument information c1 in step S103.

[0066] Next, in step S104, the acquisition unit 184 determines, based on the area of ​​interest, which camera to use to image student 100B (hereinafter referred to as "camera to use") from among cameras 111 to 115.

[0067] In step S104, the acquisition unit 184 refers to the imaging target information that indicates the imaging target for each of the cameras 111 to 115, and determines which camera from among the cameras 111 to 115 will be used to image the area of ​​interest.

[0068] Next, in step S105, the acquisition unit 184 acquires the information generated by the camera being used as target image information.

[0069] Next, in step S106, the acquisition unit 184 generates student image information a1 using the target image information.

[0070] For example, if cameras 114 and 115 are each cameras in use, the acquisition unit 184 generates student image information a1, which includes student mouth information a14 generated by camera 114 and student upper body information a15 generated by camera 115. Figure 5 shows an example of student image G3 shown by student image information a1. Student image G3 includes image G1 shown by student mouth information a14 and image G2 shown by student upper body information a15.

[0071] Next, in step S107 of Figure 4, the transmission unit 185 transmits student performance information a, which includes student image information a1 and student sound information a2, from the communication unit 160 to the teacher guidance system 200.

[0072] The teacher instruction system 200 operates in the same way as the student instruction system 100, transmitting teacher performance information b to the student instruction system 100.

[0073] Figure 6 is a diagram illustrating the operation in which the student instruction system 100 outputs a teacher image and teacher performance sound based on teacher performance information b.

[0074] In step S201, the communication unit 160 receives teacher performance information b. Teacher performance information b includes teacher image information b1 and teacher sound information b2.

[0075] Next, in step S202, the output control unit 186 displays the teacher image based on the teacher image information b1 on the display unit 130.

[0076] Next, in step S203, the output control unit 186 outputs the teacher's performance sound based on the teacher's sound information b2 from the speaker 140. Note that step S203 may be executed before step S202.

[0077] The teacher instruction system 200 operates in the same way as the student instruction system 100, displaying student images based on student image information a1 and outputting student performance sounds based on student sound information a2.

[0078] According to this embodiment, depending on the type of instrument (depending on the instrument), the image of the performer (student or teacher) necessary for instruction on playing the instrument can be identified. Furthermore, this embodiment can transmit the image of the performer necessary for instruction to the recipient. Therefore, even if teacher 200B is in a different room from the room where student 100B is playing instrument 100A, teacher 200B can observe the image of student 100B necessary for instruction on playing instrument 100A. Even if student 100B is in a different room from the room where teacher 200B is playing instrument 200A, student 100B can view an image of teacher 200B playing, which serves as an example of how to play instrument 200A.

[0079] The decision unit 183 of the student instruction system 100 may determine the point of focus using teacher instrument information d1 or d2 instead of student instrument information c1 or c2. For example, the communication unit 160 of the teacher instruction system 200 transmits teacher instrument information d1 or d2 to the student instruction system 100. The decision unit 183 of the student instruction system 100 obtains teacher instrument information d1 or d2 via the communication unit 160 of the student instruction system 100. In this case, the identification unit 181 and the trained model 182 can be omitted in the student instruction system 100.

[0080] The decision unit 183 of the teacher instruction system 200 may determine the area of ​​focus using student instrument information c1 or c2 instead of teacher instrument information d1 or d2. For example, the communication unit 160 of the student instruction system 100 transmits student instrument information c1 or c2 to the teacher instruction system 200. The decision unit 183 of the teacher instruction system 200 obtains student instrument information c1 or c2 via the communication unit 160 of the teacher instruction system 200. In this case, the identification unit 181 and the trained model 182 can be omitted in the teacher instruction system 200.

[0081] B: Modification The following describes the variations in the above-described embodiment. Two or more embodiments may be arbitrarily selected from the following embodiments and combined as appropriate, provided they do not contradict each other.

[0082] B1: First variation In the embodiments described above, the types of instruments are not limited to piano and flute; there may be two or more types. For example, the types of instruments may be two or more from among piano, flute, electronic organ (registered trademark), violin, guitar, saxophone, and drums. Piano, flute, electronic organ, violin, guitar, saxophone, and drums are each examples of instruments.

[0083] Figure 7 shows an example of a correspondence table Ta1 used when the instrument types are piano, flute, electronic organ, violin, guitar, saxophone, and drums.

[0084] For example, in an Electone lesson, the student operates the Electone as follows: The student faces the Electone in a posture of their choice. The student operates the upper and lower keyboards of the Electone with each finger of their hand. The student operates the pedal keyboard of the Electone with their feet (toes and heels). The student operates the expression pedal of the Electone with their right foot.

[0085] In Electone lessons, the teacher focuses on each finger of the student's hand, both of the student's feet (especially the right foot), and the student's entire body (e.g., posture) in order to instruct the student. The teacher instructs the student by showing them at least one of the following: each finger of the teacher's hand, both of the teacher's feet (especially the right foot), and the teacher's entire body (e.g., posture).

[0086] Therefore, in correspondence table Ta1, the instrument type "Electone" is associated with body parts "each finger of the hand, both feet, the right foot, and the whole body."

[0087] In violin lessons, students operate the violin as follows: They support the violin with their chin, shoulder, and left hand, and hold the bow with their right hand. They press the strings of the violin with the fingers of their left hand. They play the violin while changing the angle of the violin relative to them, the angle of the bow relative to the violin, and the position of their left-hand fingers relative to the strings.

[0088] In violin lessons, the teacher focuses on the student's upper body (the student's position relative to the violin) and the student's left hand in order to instruct the student. The teacher instructs the student by showing at least one of the teacher's upper body (the teacher's position relative to the violin) and the teacher's left hand to the student.

[0089] Therefore, in correspondence table Ta1, the instrument type "violin" is associated with the body part "upper body and left hand".

[0090] In guitar lessons, the student presses down on the guitar strings with their left hand and plucks them with their right hand. The teacher focuses on the student's right and left hands to guide them. The teacher guides the student by showing them their right hand and at least one of their left hands.

[0091] Therefore, in correspondence table Ta1, the instrument type "guitar" is associated with the body parts "left hand and right hand".

[0092] In saxophone lessons, the student positions the saxophone near their upper body, holds the reed in their mouth, and operates the keys and levers with their fingers. The teacher focuses on the student's mouth and upper body (e.g., how they hold the reed, how they position their mouth on the mouthpiece, their posture, the angle between the student and the saxophone, and their fingering) to guide the student. The teacher guides the student by showing them at least one of their own mouth and upper body.

[0093] Therefore, in correspondence table Ta1, the instrument type "saxophone" is associated with the body part "mouth and upper body".

[0094] In drum lessons, students play the drums using their own hands and feet. The instructor focuses on the student's hands, feet, and entire body to guide them (for example, by showing them the timing of hand and foot movements). The instructor also guides the student by demonstrating their own hand and foot movements and their own body movements to the student.

[0095] Therefore, in correspondence table Ta1, the instrument type "drum" is associated with the body parts "hands, feet, and whole body".

[0096] Furthermore, each of the student training system 100 and the teacher instruction system 200 has a camera for imaging the body parts shown in the corresponding table Ta1.

[0097] According to the first modified example, depending on the type of instrument, which is different from either the piano or the flute, the image of the performer necessary for teaching how to play the instrument can be switched, and the image can be transmitted to the recipient.

[0098] B2: Second variation In the above-described embodiment and the first modified example, the determination unit 183 may determine the points of interest on the performer's body without using either the correspondence table Ta or Ta1. For example, the determination unit 183 may determine the points of interest on the performer's body by using a trained model that has learned the relationship between the type of instrument and the body part.

[0099] Figure 8 shows a student training system 101 that includes a trained model 187 that has learned the relationship between the type of instrument and the part of the body.

[0100] The trained model 187 is composed of a neural network. For example, the trained model 187 is composed of a deep neural network. The trained model 187 may be composed of a convolutional neural network, for example. The trained model 187 may be composed of a combination of multiple types of neural networks. The trained model 187 may have additional elements such as Self-Attention. The trained model 187 may not be composed of a neural network, but rather of a Hidden Markov Model or a Support Vector Machine.

[0101] The processing unit 180 functions as a trained model 187 based on a combination of an arithmetic program that defines an operation to identify an output Y1 from an input X1, and a plurality of variables K2. The plurality of variables K2 are identified by machine learning using a plurality of training data T2. The training data T2 includes a combination of information indicating the type of instrument (training input data) and information indicating body parts (training output data). The information indicating the type of instrument in the training data T2 indicates, for example, the type of instrument shown in Figure 7. The information indicating body parts in the training data T2 indicates, for example, the body parts shown in Figure 7. In the training data T2, the combination of information indicating the type of instrument and information indicating body parts corresponds to the combination of type of instrument and body parts shown in Figure 7. Therefore, the information indicating body parts in the training data T2 indicates the part of the body of a performer using an instrument of the type indicated in the training input data of the training data T2 that is noticed by the teacher of that instrument (point of interest).

[0102] The decision unit 183 inputs student instrument information c1 or c2 into the trained model 187. Subsequently, the decision unit 183 determines the location indicated by the information output by the trained model 187 in response to the input of student instrument information c1 or c2 as the point of focus on the performer's body.

[0103] Note that multiple training data sets T2 may have only training input data and no training output data. In this case, multiple variables K2 are identified by machine learning so that the multiple training data sets T2 can be divided into multiple clusters based on the similarity of the multiple training data sets T2. Then, in the trained model 187, information indicating a body part (point of interest) appropriate for that cluster is associated with each cluster by a person. The trained model 187 identifies the cluster according to the input X1 and generates information corresponding to the identified cluster as output Y1.

[0104] According to the second modified example, the determination unit 183 can determine the body part of the performer without using either the corresponding table Ta or Ta1.

[0105] B3: Third variation In the embodiments described above and the first to second modifications, if the area of ​​interest is a part of the body (for example, both feet), the acquisition unit 184 may acquire image information indicating the area of ​​interest from whole-body image information showing the entire body of the performer.

[0106] Figure 9 shows an example of the relationship between image G11, which represents whole-body image information, and image G12, which represents a part of the performer's body. Image G12 shows both of the performer's feet as a part of the performer's body. Image G12 may also show a part of the performer's body other than both of the performer's feet.

[0107] The position of image G12 in image G11 is predetermined in pixels for each type of instrument. Therefore, the position of image G12 in image G11 can be changed according to the type of instrument. The acquisition unit 184 acquires the portion of the whole-body image information representing image G11 that is predetermined according to the type indicated by the student instrument information c1 or c2, as image information representing image G12.

[0108] The position of image G12 in image G11 does not need to be predetermined for each type of instrument. For example, the acquisition unit 184 first identifies the area of ​​interest from image G1 by using image recognition technology. Subsequently, the acquisition unit 184 acquires the area of ​​interest from the whole-body image information.

[0109] The acquisition unit 184 may use image recognition technology to determine the position of image G12 in image G11, but only for instruments where the positional relationship between the performer and the instrument is likely to change, such as flutes, violins, guitars, and saxophones. In this case, it becomes easier to acquire image information indicating the point of interest compared to a configuration where the position of image G12 in image G11 is fixed.

[0110] For instruments where the positional relationship between the performer and the instrument is not easily altered, such as pianos, electronic organs, and drums, the acquisition unit 184 acquires a portion of the whole body image information that is pre-set according to the type indicated by the student instrument information c1 or c2, as image information representing image G12. In this case, the acquisition unit 184 can easily identify the position of image G12 without using image recognition technology.

[0111] According to the third modified example, the number of cameras can be reduced compared to a configuration where multiple cameras are assigned one-to-one to multiple body parts (areas of interest).

[0112] B4: Fourth variation In the embodiments described above and the first to third modifications, the destination of the teacher performance information b is not limited to the student instruction system 100, but may also be an electronic device used by the guardian of student 100B (for example, the parent of student 100B). The electronic device may be, for example, a smartphone, tablet, or notebook personal computer. The destination of the teacher performance information b may be both the student instruction system 100 and the electronic device used by the guardian of student 100B.

[0113] According to the fourth modification, the parent of student 100B can instruct student 100B while watching a video of the teacher.

[0114] B5: Fifth variation In the embodiments described above and the first to fourth modifications, the related information relating to the type of instrument (related information regarding the instrument) is not limited to student sound information a2. The related information may also be image information showing instrument 100A (image information showing an image representing instrument 100A).

[0115] In a configuration where image information representing the instrument 100A is used as related information, the identification unit 181 identifies the instrument information (student instrument information c2) by using a trained model that has learned the relationship between information representing the instrument in the image and information representing the type of instrument represented by that image.

[0116] Figure 10 shows a student instruction system 102 that includes a trained model 188 that has learned the relationship between information showing an image of a musical instrument and information showing the type of instrument. The trained model 188 is an example of a first trained model.

[0117] The trained model 188 is composed of a neural network. For example, the trained model 188 is composed of a deep neural network. The trained model 188 may be composed of a convolutional neural network, for example. The trained model 188 may be composed of a combination of multiple types of neural networks. The trained model 188 may have additional elements such as Self-Attention. The trained model 188 may not be composed of a neural network, but rather of a Hidden Markov Model or a Support Vector Machine.

[0118] The processing unit 180 functions as a trained model 188 based on a combination of an operation program that defines an operation to determine the output Y1 from the input X1, and a plurality of variables K3. The plurality of variables K3 are identified by machine learning using a plurality of training data T3. The training data T3 includes a combination of information that shows musical instruments in images (training input data) and information that shows the type of musical instrument shown in the training input data in images (training output data).

[0119] The identification unit 181 inputs image information representing the instrument 100A into the trained model 188. Subsequently, the identification unit 181 identifies the information output by the trained model 188 in response to the input of image information representing the instrument 100A as student instrument information c2.

[0120] Note that multiple training data sets T3 may have only training input data and no training output data. In this case, multiple variables K3 are identified by machine learning so that the multiple training data sets T3 can be divided into multiple clusters based on the similarity of the multiple training data sets T3. Then, in the trained model 188, information indicating the type of musical instrument appropriate for each cluster is associated by a human. The trained model 188 identifies the cluster corresponding to the input X1 and generates information corresponding to the identified cluster as output Y1.

[0121] According to the fifth modification, image information showing the instrument 100A can be used as related information indicating the instrument.

[0122] B6: Sixth variation In the fifth modified example, the specific unit 181 may use information generated by any of the cameras 111 to 115 (hereinafter referred to as "camera image information") as image information indicating the musical instrument 100A.

[0123] Camera image information may show instrument 100A and student 100B, as well as other types of instruments different from instrument 100A. When camera image information showing multiple types of instruments is input to the trained model 188, the information output from the trained model 188 may not indicate the type of instrument 100A. For this reason, the identification unit 181 first extracts partial image information showing only instrument 100A from the camera image information. Then, the identification unit 181 inputs the partial image information to the trained model 188.

[0124] For example, the identification unit 181 first identifies a human (student 100B) from the image shown by the camera image information. Humans are easier to recognize than musical instruments. Next, the identification unit 181 identifies the object closest to the human (student 100B) in the image shown by the camera image information as the musical instrument 100A. Then, the identification unit 181 extracts partial image information from the camera image information that shows only the object identified as the musical instrument 100A. Next, the identification unit 181 inputs the partial image information into the trained model 188.

[0125] According to the sixth modification, the camera image information generated by any of the cameras 111 to 115 can be used as related information related to the type of musical instrument. Therefore, any of the cameras 111 to 115 can also be used as a device for generating related information.

[0126] B7: Seventh variation In the embodiments described above and the first to sixth modifications, the relevant information relating to the type of instrument may be musical score information showing musical scores corresponding to the type of instrument. A musical score corresponding to a type of instrument (e.g., guitar) is an example of a musical score corresponding to an instrument (e.g., guitar). A musical score is also called a musical note. Musical score information is generated, for example, by a camera that images the musical score. If any of the cameras 111 to 115 generate musical score information, any of the cameras 111 to 115 can be used as a device for generating musical score information.

[0127] The identification unit 181 identifies student instrument information c2 based on the musical score indicated by the musical score information. For example, the identification unit 181 identifies student instrument information c2 based on the type of musical score.

[0128] If the musical score indicated by the musical score information is a tablature, the identification unit 181 identifies student instrument information c2, which indicates a guitar as the type of instrument. As shown in Figure 11, tablature uses six parallel lines to represent the strings of a guitar. Therefore, if the musical score indicated by the musical score information consists of six parallel lines, the identification unit 181 determines that the musical score indicated by the musical score information is a tablature.

[0129] If the musical score indicated by the musical score information is a guitar chord chart, the identification unit 181 identifies student instrument information c2, which indicates a guitar as the type of instrument. As shown in Figure 12, a guitar chord chart represents guitar chords that follow the order of the lyrics. Therefore, if the musical score indicated by the musical score information represents guitar chords, the identification unit 181 determines that the musical score indicated by the musical score information is a guitar chord chart.

[0130] If the musical score indicated by the musical score information is a drum score, the identification unit 181 identifies student instrument information c2 which indicates drums as the type of instrument. As shown in Figure 13, a drum score represents symbols corresponding to each instrument included in a drum set. Therefore, if the musical score indicated by the musical score information represents symbols corresponding to each instrument in a drum set, the identification unit 181 determines that the musical score indicated by the musical score information is a drum score.

[0131] If the musical score indicated by the musical score information is a piano duet, the identification unit 181 identifies student instrument information c2 which indicates piano as the type of instrument. A piano duet is represented by the symbol 14a indicating a piano duet, as shown in Figure 14. Therefore, if the musical score indicated by the musical score information represents the symbol 14a indicating a piano duet, the identification unit 181 determines that the musical score indicated by the musical score information is a piano duet.

[0132] The identification unit 181 may identify student instrument information c2 based on the arrangement of notes in the musical score indicated by the musical score information. As shown in Figure 15, if the musical score indicated by the musical score information represents notes 15a indicating the simultaneous sounding of multiple notes, the identification unit 181 identifies the musical score indicated by the musical score information as being for a keyboard instrument (for example, a piano or an electronic organ). In this case, the identification unit 181 identifies student instrument information c2 indicating piano or electronic organ as the type of instrument.

[0133] If the musical score indicated by the musical score information shows a symbol that identifies the type of instrument (for example, a string representing the name of the instrument, or a code related to the type of instrument), the identification unit 181 may identify the information indicating the type of instrument identified by the symbol as student instrument information c2. For example, if the storage device 170 stores an instrument table that shows the correspondence between information indicating the type of instrument and a code related to the type of instrument, the identification unit 181 refers to the instrument table and identifies the information corresponding to the symbol shown in the musical score (information indicating the type of instrument) as student instrument information c2. In this case, the code related to the type of instrument is an example of related information. The instrument table is an example of a table that shows the correspondence between information related to the type of instrument and information indicating the type of instrument. Information related to the type of instrument is an example of reference related information about instruments. Information indicating the type of instrument is an example of reference instrument information that indicates an instrument.

[0134] The musical score information is not limited to information generated by a camera that captures images of the musical score, but may also be so-called electronic musical scores. If the electronic musical score has type data indicating the type of instrument, the identification unit 181 may identify the type data as student instrument information c2.

[0135] According to the seventh modification, musical score information can be used as related information related to the type of instrument.

[0136] B8: Eighth variation In the embodiments described above and the first to seventh modifications, if the schedule information indicating student 100B's schedule indicates the type of instrument, the schedule information may be used as related information related to the type of instrument. The schedule information may indicate the schedules of student 100B, teacher 200B, the student's room in the music classroom, or the teacher's room in the music classroom, as long as it indicates a combination of the type of instrument and the instruction schedule for that type of instrument. The combination of the type of instrument (e.g., piano) and the instruction schedule for that type of instrument (e.g., piano) is an example of the combination of the instrument (e.g., piano) and the instruction schedule for that instrument (e.g., piano).

[0137] Figure 16 is a diagram showing an example of a schedule as indicated by the schedule information. In Figure 16, the type of instrument being taught (piano, flute, or violin) is shown for each lesson time slot. First, the identification unit 181 uses the schedule information to identify the lesson time slot that includes the current time. Next, the identification unit 181 identifies the type of instrument being taught that corresponds to the identified time slot. Subsequently, the identification unit 181 identifies the information indicating the identified type of instrument being taught as student instrument information c2.

[0138] Figure 17 shows another example of a schedule as shown by the schedule information. In Figure 17, the type of instrument being taught is shown for each lesson date. The identification unit 181 first uses the schedule information to identify the type of instrument being taught corresponding to the current date. Subsequently, the identification unit 181 identifies the information indicating the identified type of instrument being taught as student instrument information c2.

[0139] According to the eighth modification, schedule information can also be used as related information related to the type of instrument.

[0140] B9: Ninth Revision In the embodiments described above and the first to eighth modified examples, the determination unit 183 may determine the points of interest based on student instrument information c1 or c2 and student sound information a2.

[0141] In piano lessons, teacher 200B often focuses on the movement of each finger of student 100B's hand during the faster sections of the piece being taught. Therefore, in piano lessons, if the student's performance indicated by student sound information a2 represents the section immediately preceding a fast section of the piece, the determination unit 183 determines only the fingers of the hand as the area of ​​focus. Subsequently, if the student's performance indicated by student sound information a2 represents the section immediately following a fast section, the determination unit 183 determines the fingers of the performer's hand, both feet of the performer, and the performer's entire body as the areas of focus.

[0142] In this case, the storage device 170 stores musical score data indicating the portion immediately preceding the fast part of the song and the portion immediately following the fast part. The determination unit 183 generates note data indicating the student's performance based on the student sound information a2. If the note data matches the portion immediately preceding the fast part of the song in the musical score data, the determination unit 183 determines that the student's performance indicates the portion immediately preceding the fast part of the song. The determination unit 183 may also determine that the student's performance indicates the preceding portion if the degree of agreement between the note data and the preceding portion is 90% or higher. The first threshold is not limited to 90% and can be changed as appropriate. If the note data matches the portion immediately following the fast part of the song in the musical score data, the determination unit 183 determines that the student's performance indicates the portion immediately following the fast part of the song. The determination unit 183 may also determine that the student's performance indicates the immediately following portion if the degree of agreement between the note data and the immediately following portion is 90% or higher. The second threshold is not limited to 90% and can be changed as needed.

[0143] Regarding the piano, the timing of the change in the point of focus is not limited to the timing when the student's performance indicates the section immediately preceding or immediately following a fast section of the piece; it can be changed as appropriate. Regarding the piano, the transition of the point of focus is not limited to the transitions described above; it can be changed as appropriate.

[0144] For instruments other than the piano, the determination unit 183 may also determine the points of interest based on student instrument information c1 or c2 and student sound information a2.

[0145] For example, in flute lessons, teacher 200B often focuses on the shape of student 100B's mouth during the beginning of a piece. Therefore, in flute lessons, when the student's performance indicated by student sound information a2 represents the beginning of a piece, the determination unit 183 determines only the mouth as the area of ​​focus. Subsequently, when the student's performance indicated by student sound information a2 represents the section immediately following the beginning of the piece, the determination unit 183 determines the performer's mouth and upper body as the areas of focus.

[0146] In this case, the storage device 170 stores musical score data indicating the beginning portion of the song and the portion immediately following the beginning portion. The determination unit 183 generates note data indicating the student's performance based on the student sound information a2. If the note data matches the beginning portion of the song in the musical score data, the determination unit 183 determines that the student's performance indicates the beginning portion of the song. The determination unit 183 may also determine that the student's performance indicates the beginning portion if the degree of agreement between the note data and the beginning portion is 90% or higher. The third threshold is not limited to 90% and can be changed as appropriate. If the note data matches the portion immediately following the beginning portion of the song in the musical score data, the determination unit 183 determines that the student's performance indicates the portion immediately following the beginning portion of the song. The determination unit 183 may also determine that the student's performance indicates the portion immediately following the beginning portion if the degree of agreement between the note data and the portion immediately following the beginning portion is 90% or higher. The fourth threshold is not limited to 90% and can be changed as appropriate.

[0147] Regarding the flute, the timing of the change in the point of focus can be changed as appropriate, not limited to the timing when the student's performance indicates the beginning of the piece, or the timing when the student's performance indicates the section immediately following the beginning of the piece. Regarding the flute, the transition of the point of focus can be changed as appropriate, not limited to the transitions described above.

[0148] The decision unit 183 may determine the points of interest using a trained model that has learned the relationship between information including instrument type information indicating the type of instrument, instrument sound information indicating the sound output from the instrument of the type indicated by the instrument type information, and information indicating points of interest on the performer's body. The instrument type information is an example of training instrument information indicating an instrument. The instrument sound information is an example of training sound information indicating the sound output from the instrument indicated by the training instrument information. The information including instrument type information and instrument sound information is an example of training input information. The information indicating points of interest on the performer's body indicates the points of interest on the performer's body that are focused on by the teacher of the instrument, when the instrument outputs the sound indicated by the instrument sound information from the instrument of the type indicated by the instrument type information. The information indicating points of interest on the performer's body is an example of training output information indicating the points of interest on the body of a performer who plays an instrument that is indicated by the training instrument information and outputs the sound indicated by the training sound information.

[0149] Figure 18 shows a student instruction system 103 that includes a trained model 189 that has learned the correspondence between combinations of instrument type information and instrument sound information, and information indicating points of interest. The trained model 189 is an example of a second trained model.

[0150] The trained model 189 is composed of a neural network. For example, the trained model 189 is composed of a deep neural network. The trained model 189 may be composed of a convolutional neural network, for example. The trained model 189 may be composed of a combination of multiple types of neural networks. The trained model 189 may have additional elements such as Self-Attention. The trained model 189 may not be composed of a neural network, but rather of a Hidden Markov Model or a Support Vector Machine.

[0151] The processing unit 180 functions as a trained model 189 based on a combination of an operation program that defines an operation to identify an output Y1 from an input X1, and a plurality of variables K4. The plurality of variables K4 are identified by machine learning using a plurality of training data T4. The training data T4 includes a combination of instrument type information and instrument sound information (training input data) and point of focus information indicating points of focus on the body (training output data). The point of focus information indicates points of focus on the body of a performer who outputs the sound indicated by the instrument sound information from the type of instrument indicated by the instrument type information, and which are points of focus by the teacher of that instrument.

[0152] Instrument sound information is used for each measure of the piece being performed. Instrument sound information may be used not only for each measure, but for example, every four measures. The point of interest information (training output data) indicates the point of interest on the body of the performer using the instrument indicated by the instrument type information, during the performance of the measure immediately following the measure indicated by the instrument sound information in the training input data.

[0153] The decision unit 183 inputs a pair of student instrument information c1 or c2 and student sound information a2 to the trained model 189 for each measure. The decision unit 183 generates note data representing the student's performance based on the student sound information a2, and identifies a measure in the student sound information a2 based on the arrangement of this note data. Subsequently, the decision unit 183 determines the location indicated by the information output by the trained model 189 in response to the input of the pair of student instrument information c1 or c2 and student sound information a2 as a point of interest.

[0154] Note that multiple training data sets T4 may have only training input data and no training output data. In this case, multiple variables K4 are identified by machine learning so that the multiple training data sets T4 can be divided into multiple clusters based on the similarity of the multiple training data sets T4. Then, in the trained model 189, information indicating a body part (point of interest) appropriate for that cluster is associated with each cluster by a person. The trained model 189 identifies the cluster according to the input X1 and generates the information corresponding to the identified cluster as output Y1.

[0155] According to the ninth modification, the images necessary for teaching the type of instrument indicated by student instrument information c1 or c2 can be identified based on the sound being played.

[0156] B10: 10th variation In the ninth modified example, the student instruction system 100 and the teacher instruction system 200 may be used for instruction in playing one type of instrument (for example, the piano). The one type of instrument is not limited to the piano and can be changed as appropriate. In this case, the decision unit 183 determines the points of focus on the performer's body based on the student sound information a2. For example, the decision unit 183 inputs the student sound information a2 for each measure to a trained model (trained model) that has been trained on training data, which is a combination of instrument sound information (training input data) and points of focus information (training output data) indicating points of focus on the body. In this case, the points of focus information (training output data) indicating points of focus on the body indicates the points of focus on the body of a performer using an instrument that outputs the sound indicated by the instrument sound information (training input data), as the points of focus for that instrument. Subsequently, the decision unit 183 determines the points of focus as indicated by the information output by the trained model in response to the input of the student sound information a2. According to the tenth modified example, images necessary for instrument instruction can be identified based on the sound of the performance.

[0157] B11: 11th Variant In the embodiments described above and the first to tenth modified examples, the determination unit 183 may determine the points of interest on the body based on the correspondence between student sound information a2 and musical score information showing the musical score of the song. The correspondence between student sound information a2 and musical score information is an example of the relationship between student sound information a2 and musical score information.

[0158] The determination unit 183 determines the degree of agreement between the sound indicated by the student sound information a2 and the sound represented in the musical score indicated by the musical score information.

[0159] For example, in piano lessons, if the student's playing is inconsistent, teacher 200B often focuses on the movement of each finger of student 100B's hand. In piano lessons, if the degree of agreement is below a threshold, the determination unit 183 determines that only the fingers of the hand are the area of ​​focus. If the degree of agreement is above a threshold, the determination unit 183 determines that the fingers of the performer's hand, both feet of the performer, and the performer's entire body are the areas of focus.

[0160] In flute lessons, if a student's playing is inconsistent, teacher 200B often focuses on student 100B's mouth and upper body. In flute lessons, if the degree of agreement is below a threshold, the determination unit 183 determines the mouth and upper body as areas of focus. If the degree of agreement is above a threshold, the determination unit 183 determines the performer's upper body as areas of focus.

[0161] The decision unit 183 may determine the point of interest using a trained model that has learned the relationship between information including output sound information indicating the sound output from the instrument, score-related information indicating the musical score, and information indicating the body parts of the performer. The output sound information is an example of training sound information indicating the sound output from the instrument. The score-related information is an example of training score information indicating the musical score. The information including the output sound information and the score-related information is an example of training input information. The information indicating the body parts of the performer indicates the point of interest (point of interest) on the performer's body that outputs the sound indicated by the output sound information from the instrument according to the musical score indicated by the score-related information. The information indicating the body parts of the performer is an example of training output information that indicates the point of interest on the body of the performer that plays an instrument that outputs the sound indicated by the training sound information according to the musical score indicated by the training score information.

[0162] Figure 19 shows a student training system 104 that includes a trained model 190 that has learned the relationship between pairs of output sound information and musical score-related information, and information indicating the points of focus on the performer's body. The trained model 190 is an example of a third trained model.

[0163] The trained model 190 is composed of a neural network. For example, the trained model 187 is composed of a deep neural network. The trained model 190 may be composed of a convolutional neural network, for example. The trained model 190 may be composed of a combination of multiple types of neural networks. The trained model 190 may have additional elements such as Self-Attention. The trained model 190 may not be composed of a neural network, but rather of a Hidden Markov Model or a Support Vector Machine.

[0164] The processing unit 180 functions as a trained model 190 based on a combination of an operation program that defines an operation to identify output Y1 from input X1, and a plurality of variables K5. The plurality of variables K5 are identified by machine learning using a plurality of training data T5. The training data T5 is a combination of a pair of output sound information and musical score relationship information (training input data) and attention point information indicating points of interest in the body (training output data). The attention point information (training output data) indicates points on the body of a performer who outputs the sound indicated by the output sound information from the instrument according to the musical score indicated by the musical score relationship information, that are noticed by the teacher of the instrument.

[0165] The output sound information is used for each measure of the music being played. The output sound information is not limited to every measure; for example, it may be used every four measures. The focus point information (training output data) indicates the focus point in the measure immediately following the measure indicated by the output sound information in the training input data.

[0166] The decision unit 183 inputs a pair of student sound information a2 and musical score information to the trained model 190 for each measure. The pair of student sound information a2 and musical score information is an example of input information that includes sound information and musical score information. The decision unit 183 generates note data representing the student's performance based on the student sound information a2, and identifies one measure in the student sound information a2 based on the arrangement of this note data. Subsequently, the decision unit 183 determines the location indicated by the information output by the trained model 190 in response to the input of the pair of student sound information a2 and musical score information as a point of interest.

[0167] Note that multiple training data sets T5 may have only training input data and no training output data. In this case, multiple variables K5 are identified by machine learning so that the multiple training data sets T5 can be divided into multiple clusters based on the similarity of the multiple training data sets T5. Then, in the trained model 190, information indicating a body part (point of interest) appropriate for that cluster is associated with each cluster by a person. The trained model 190 identifies the cluster according to the input X1 and generates the information corresponding to the identified cluster as output Y1.

[0168] According to the 11th variation, the images necessary for instruction can be switched according to the correspondence between the student's performance sound and the musical score.

[0169] B12: Twelfth variation In the embodiments described above and the first to eleventh modifications, the determination unit 183 of the student instruction system 100 may further determine the points of interest on the body based on written information. The written information indicates notes written about the performance. The notes may be shown in text or in symbols. The written information is an example of cautionary information indicating notes about the performance.

[0170] For example, the decision unit 183 of the student instruction system 100 determines the points of focus based on the teacher's written information. The teacher's written information indicates notes written on the musical score by the teacher 200B. The teacher's written information is generated by one of the cameras 111 to 115 in the teacher guidance system 200, which images the musical score on which the notes are written. The communication unit 160 of the teacher guidance system 200 transmits the teacher's written information to the student instruction system 100. The decision unit 183 of the student instruction system 100 receives the teacher's written information via the communication unit 160 of the student instruction system 100. The storage device 170 of the student instruction system 100 stores a note table in advance, which shows the correspondence between notes and body parts. The decision unit 183 of the student instruction system 100 further determines the body parts corresponding to the notes indicated by the teacher's written information in the note table as points of focus.

[0171] The decision unit 183 of the student training system 100 may determine the points of focus based on the location of the notes in the musical score. In this case, the memory device 170 of the student training system 100 stores in advance a position table that shows the correspondence between the location in the musical score and the body part. The decision unit 183 of the student training system 100 further determines the body part corresponding to the location of the notes in the musical score as the point of focus in the position table.

[0172] Notes may be written on a different object than the musical score (for example, on a piece of paper, notebook, or whiteboard).

[0173] According to the 12th variation, points of interest can be added based on the notes written about the performance.

[0174] B13: 13th variation In the embodiments described above and the first to twelfth modifications, the determination unit 183 of the student instruction system 100 may further determine the points of interest on the body based on performer information relating to the performer. Performer information is, for example, the identification information of teacher 200B.

[0175] In instrument instruction, the areas of focus may differ depending on the instructor (200B). For example, in piano instruction, instructor 200B1 may focus on each finger, both feet, and the entire body of student 100B, as well as student 100B's right arm. In contrast, instructor 200B1 may focus on each finger, both feet, and the entire body of student 100B, as well as student 100B's left arm. Therefore, the decision unit 183 of the student instruction system 100 further determines the areas of focus based on the instructor's identification information (e.g., identification code).

[0176] The identification information of teacher 200B is input from the operation unit 150 by a user, such as student 100B. The identification information of teacher 200B may also be transmitted from the teacher instruction system 200 to the student instruction system 100. The storage device 170 of the student instruction system 100 stores in advance an identification information table that shows the correspondence between the identification information of teacher 200B and body parts. The determination unit 183 of the student instruction system 100 further determines the body parts corresponding to the identification information of teacher 200B in the identification information table as points of interest.

[0177] The performer information is not limited to identification information for teacher 200B; for example, it could be motion information indicating the movements of teacher 200B. For example, one of the cameras 111-115 in the teacher instruction system 200 generates motion information by imaging teacher 200B. The communication unit 160 of the teacher instruction system 200 transmits the motion information to the student instruction system 100. The decision unit 183 of the student instruction system 100 receives the motion information via the communication unit 160 of the student instruction system 100. The storage device 170 of the student instruction system 100 stores a motion table in advance that shows the correspondence between a person's movements and parts of the body. The decision unit 183 of the student instruction system 100 further determines the parts of the body corresponding to the movements indicated by the motion information in the motion table as points of interest. Therefore, teacher 200B can specify points of interest according to their movements. The performer information may be identification information for student 100B, or motion information indicating the movements of student 100B. In this case, the determination unit 183 can determine the points of interest according to student 100B.

[0178] According to the 13th modification, body parts of the performer can be added based on performer information about the performer.

[0179] B14: 14th variation In the embodiments described above and the first to thirteenth modifications, the touch panel operation unit 150 may have a user interface as shown in Figure 20 for receiving student instrument information c1. Touching the piano button 151 signifies the input of student instrument information c1 indicating piano as the instrument type. Touching the flute button 152 signifies the input of student instrument information c1 indicating flute as the instrument type. The user interface for receiving student instrument information c1 is not limited to the user interface shown in Figure 20. According to the fourteenth modification, the user can intuitively input student instrument information c1.

[0180] B15: 15th variation In the embodiments described above and the first to fourteenth modified examples, the communication unit 160 of the teacher instruction system 200 may transmit teacher instrument information d1 or d2 to the student instruction system, and the decision unit 183 of the student instruction system may determine the points of interest based on the teacher instrument information d1 or d2. Alternatively, the communication unit 160 of the student instruction system may transmit student instrument information c1 or c2 to the teacher instruction system, and the decision unit 183 of the teacher instruction system may determine the points of interest based on the student instrument information c1 or c2. Furthermore, the configuration of the teacher instruction system 200 may be the same as any one of the configurations of the student instruction systems 101 to 105.

[0181] B16: 16th variation In the embodiments described above and the first to fifteenth modifications, the processing unit 180 may generate a trained model 182.

[0182] Figure 21 shows a student training system 105 according to the 16th modified example. The student training system 105 differs from the student training system 104 shown in Figure 19 in that it has a learning processing unit 191. The learning processing unit 191 is implemented by a processing unit 180 that executes a machine learning program. The machine learning program is stored in a storage device 170.

[0183] Figure 22 shows an example of the learning processing unit 191. The learning processing unit 191 includes a data acquisition unit 192 and a training unit 193. The data acquisition unit 192 acquires multiple training data T1. For example, the data acquisition unit 192 acquires multiple training data T1 via the operation unit 150 or the communication unit 160. If the storage device 170 stores multiple training data T1, the data acquisition unit 192 acquires the multiple training data T1 from the storage device 170.

[0184] The training unit 193 generates a trained model 182 by performing a process using multiple training data T1 (hereinafter referred to as the "learning process"). The learning process is supervised machine learning using multiple training data T1. The training unit 193 changes the target model 182a to the trained model 182a by training the target model 182a using multiple training data T1.

[0185] The model to be trained 182a is generated by a processing unit 180 that uses a set of provisional variables K1 and an arithmetic program. The set of provisional variables K1 are stored in a memory device 170. The model to be trained 182a differs from the trained model 182 in that it uses a set of provisional variables K1. The model to be trained 182a generates information (output data) according to the information (input data) that is input.

[0186] The training unit 193 identifies the value of the loss function L, which represents the error between the output data generated by the target model 182a when the input data in the training data T1 is input to the target model 182a, and the output data in the training data T1. The training unit 193 updates several provisional variables K1 to reduce the value of the loss function L. The training unit 193 performs the process of updating several provisional variables K1 for each of the training data T1 sets. Upon completion of training by the training unit 193, the multiple variables K1 are finalized. The target model 182a after training by the training unit 193, i.e., the trained model 182, outputs statistically valid output data for unknown input data.

[0187] Figure 23 shows an example of a learning process. For example, the learning process may be initiated by a user's instruction.

[0188] In step S301, the data acquisition unit 192 acquires unacquired training data T1 from among multiple training data T1. Subsequently, in step S302, the training unit 193 trains the model to be learned 182a using the training data T1. In step S302, the training unit 193 updates several provisional variables K1 so that the value of the loss function L, which is determined using the training data T1, is reduced. For example, backpropagation can be used to update the several provisional variables K1 according to the value of the loss function L.

[0189] Next, in step S303, the training unit 193 determines whether the termination condition for the learning process has been met. The termination condition is, for example, that the value of the loss function L falls below a predetermined threshold, or that the amount of change in the value of the loss function L falls below a predetermined threshold. If the termination condition is not met, the process returns to step S301. For this reason, the acquisition of training data T1 and the updating of provisional variables K1 using the training data T1 are repeated until the termination condition is met. If the termination condition is met, the learning process ends.

[0190] The learning processing unit 191 may be implemented in a processing unit different from the processing unit 180. The processing unit different from the processing unit 180 includes at least one computer.

[0191] The data acquisition unit 192 may acquire one or more types of training data from among four types of training data, such as training data T2, T3, T4, and T5, which are different from the training data T1. The training unit 193 trains a target model according to the type of training data acquired by the data acquisition unit 192. The target model according to the training data T2 is a target model generated by the processing unit 180 using a provisional set of variables K2 and a calculation program. The target model according to the training data T3 is a target model generated by the processing unit 180 using a provisional set of variables K3 and a calculation program. The target model according to the training data T4 is a target model generated by the processing unit 180 using a provisional set of variables K4 and a calculation program. The target model according to the training data T5 is a target model generated by the processing unit 180 using a provisional set of variables K5 and a calculation program.

[0192] The data acquisition unit 192 may be provided for each of the multiple types of training data. In this case, each data acquisition unit 192 acquires the corresponding multiple training data.

[0193] The training unit 193 may be provided for each of the multiple types of training data. In this case, each training unit 193 uses the corresponding multiple training data to train a target model that corresponds to the multiple training data.

[0194] According to the 16th modification, the learning processing unit 241 can generate at least one trained model.

[0195] B17: 17th variation In the embodiments described above and the first to sixteenth modifications, the processing unit 180 may function only as a determination unit 183 and an acquisition unit 184, as shown in Figure 24. The determination unit 183 shown in Figure 24 determines a point of interest from the body of a performer using an instrument of the type indicated by the instrument information, based on the instrument information indicating the type of instrument. The acquisition unit 184 shown in Figure 24 acquires image information representing an image of the point of interest determined by the determination unit 183. According to the seventeenth modification, it is possible to identify an image of a performer necessary for teaching how to play an instrument, depending on the type of instrument.

[0196] B18: 18th variation In the 17th modified example, the determination unit 183 shown in Figure 24 may determine a point of interest from the body of a performer using an instrument based on sound information indicating the sound output from the instrument, rather than on instrument information indicating the type of instrument. Furthermore, in the 17th modified example, the acquisition unit 184 shown in Figure 24 may acquire image information representing the image of the point of interest determined by the determination unit 183 based on sound information indicating the sound output from the instrument. According to the 18th modified example, it is possible to identify an image of a performer necessary for teaching how to play an instrument, depending on the sound output from the instrument.

[0197] C: The form understood from the above-described form From at least one of the above-described forms, the following characteristics can be identified.

[0198] C1: First aspect An information processing method according to an aspect of this disclosure (the first aspect) is an information processing device executed by a computer, which determines a point of interest from the body of a performer playing an instrument indicated by the instrument information, based on instrument information indicating an instrument, and acquires image information representing the determined point of interest. According to this aspect, depending on the instrument, an image of a performer necessary for teaching how to play the instrument can be identified.

[0199] C2: Second aspect In the example of the first embodiment (second embodiment), the acquired image information is further transmitted to an external device. According to this embodiment, images of performers necessary for teaching how to play musical instruments can be transmitted to an external device.

[0200] C3: Third aspect In an example of the first or second embodiment (third embodiment), further identifying the instrument information and determining the area of ​​interest using the relevant information relating to the instrument includes determining the area of ​​interest based on the identified instrument information. According to this embodiment, based on the relevant information relating to the instrument, images of performers necessary for teaching how to play an instrument can be identified.

[0201] C4: Fourth aspect In the example of the third embodiment (fourth embodiment), the related information is information indicating the sound output by the instrument, information indicating an image representing the instrument, information indicating a musical score corresponding to the instrument, or information indicating a combination of the instrument and the instruction schedule for that instrument. According to this embodiment, various types of information can be used as related information.

[0202] C5: Fifth aspect In an example of the third or fourth embodiment (fifth embodiment), identifying the instrument information includes inputting the related information to a first trained model that has learned the relationship between the related learning information about the instrument and the learning instrument information indicating the instrument identified from the related learning information, and identifying the information output by the first trained model in accordance with the related information as the instrument information. According to this embodiment, since the instrument information is identified using a trained model, the instrument information can indicate the instrument played by the performer with high accuracy.

[0203] C6: Sixth aspect In the example of the fifth embodiment (sixth embodiment), the related information and the learning related information indicate the sound output by the instrument, and the learning instrument information indicates the instrument that outputs the sound indicated by the learning related information. According to this embodiment, the instrument can be identified based on the sound it outputs.

[0204] C7: Seventh aspect In the example of the fifth embodiment (seventh embodiment), the related information and the learning related information represent images of the musical instruments, and the learning instrument information represents the musical instruments represented by the images shown in the learning related information. According to this embodiment, musical instruments can be identified based on the images representing the musical instruments.

[0205] C8: 8th form In an example of the third embodiment (eighth embodiment), identifying the instrument information includes identifying the reference instrument information corresponding to the related information by referring to a table showing the correspondence between the related reference information for the instrument and the reference instrument information representing the instrument. According to this embodiment, instrument information can be identified without using a trained model.

[0206] C9: 9th aspect In any example of the first to eighth embodiments (ninth embodiment), determining the area of ​​interest includes determining the area of ​​interest based on sound information indicating the sound output from the instrument as indicated by the instrument information, and the instrument information. According to this embodiment, based on the sound output from the instrument, an image of the performer necessary for teaching how to play an instrument can be identified.

[0207] C10: Tenth aspect In an example of the ninth embodiment (tenth embodiment), determining the area of ​​interest involves inputting the input information, including the instrument information and the sound information, to a second trained model that has learned the relationship between learning input information, including learning instrument information indicating the instrument, and learning sound information indicating the sound output from the instrument indicated by the learning instrument information, and learning output information indicating a point of interest on the body of a performer playing the instrument indicated by the learning instrument information and the sound output by the learning sound information; and determining the area of ​​interest based on the output information output by the second trained model in response to the input information. According to this embodiment, since the area of ​​interest is identified using the trained model, the image of the performer necessary for teaching how to play an instrument can be identified with high accuracy based on the sound output from the instrument.

[0208] C11: Eleventh aspect An information processing method according to an aspect of this disclosure (the 11th aspect) is an information processing method performed by a computer, which determines a point of interest from the body of a performer playing a musical instrument based on sound information indicating the sound output from the musical instrument, and acquires image information indicating the image of the determined point of interest. According to this aspect, an image of a performer necessary for teaching how to play a musical instrument can be identified according to the sound output from the musical instrument.

[0209] C12: Twelfth aspect In an example of the ninth or eleventh embodiment (twelfth embodiment), determining the point of interest includes determining the point of interest based on the relationship between musical score information showing the musical score and the sound information. According to this embodiment, based on the relationship between the musical score information and the sound information, an image of a performer necessary for teaching how to play an instrument can be identified.

[0210] C13: 13th aspect In an example of the 11th embodiment (13th embodiment), determining the point of interest involves inputting input information, including musical score information and the sound information, to a third trained model that has learned the relationship between learning input information, including learning sound information indicating the sound output from the instrument and learning musical score information indicating the musical score, and learning output information, which indicates a point of interest on the body of a performer who plays an instrument that outputs the sound indicated by the learning sound information according to the musical score indicated by the learning musical score information; and determining the point of interest based on the output information output by the third trained model in response to the input information. According to this embodiment, since the point of interest is identified using a trained model, images of performers necessary for teaching instrument playing can be identified with high accuracy.

[0211] C14: 14th aspect In any example of the first to thirteenth embodiments (fourteenth embodiment), determining the area of ​​interest includes determining the area of ​​interest based on cautionary information indicating precautions regarding performance. According to this embodiment, the image of the performer necessary for instruction on playing an instrument can be switched according to the precautions regarding performance.

[0212] C15: 15th aspect In any example of the first to fourteenth embodiments (fifteenth embodiment), determining the area of ​​interest includes determining the area of ​​interest based on performer information relating to the performer. According to this embodiment, the images of the performer necessary for teaching how to play an instrument can be switched according to the performer information relating to the performer.

[0213] C16: 16th aspect An information processing system according to an aspect of this disclosure (the 16th aspect) includes a determination unit that determines a point of interest from the body of a performer playing an instrument indicated by instrument information, based on instrument information indicating the instrument, and an acquisition unit that acquires image information representing an image of the point of interest determined by the determination unit. According to this aspect, depending on the instrument, an image of a performer necessary for teaching how to play the instrument can be identified.

[0214] C17: 17th aspect An information processing system according to an aspect of this disclosure (Aspect 17) includes a determination unit that determines a point of interest from the body of a performer using a musical instrument based on sound information indicating the sound output from the instrument, and an acquisition unit that acquires image information representing an image of the point of interest determined by the determination unit. According to this aspect, an image of a performer necessary for teaching how to play a musical instrument can be identified according to the sound output from the instrument. [Explanation of symbols]

[0215] 1...Information provision system, 100...Student training system, 100A...Musical instrument, 100B...Student, 111~115...Camera, 120...Microphone, 130...Display unit, 140...Speaker, 150...Operation unit, 160...Communication unit, 170...Storage device, 180...Processing device, 181...Identification unit, 182...Learned model, 182a...Model to be learned, 183...Decision unit, 184...Acquisition unit, 185...Transmission unit, 186...Output control unit, 187~190...Learned model, 191...Learning processing unit, 192...Data acquisition unit, 193...Training unit, 200...Teacher guidance system, 200A...Musical instrument, 200B...Teacher.

Claims

1. A determination unit that uses a table showing the correspondence between points of interest on the body of a musician playing an instrument and information about the instructor who teaches the musician to determine the points of interest based on the information about the instructor, The acquisition unit acquires image information representing the image of the area of ​​interest determined by the aforementioned determination unit, Includes, The information relating to the said leader includes the identification information of the said leader. Information processing system.

2. A determination unit that uses a table showing the correspondence between points of interest on the body of a musician playing an instrument and information about the instructor who teaches the musician to determine the points of interest based on the information about the instructor, The acquisition unit acquires image information representing the image of the area of ​​interest determined by the aforementioned determination unit, Includes, The information relating to the instructor includes motion information indicating the instructor's movements. Information processing system.

3. A determination unit that uses a table showing the correspondence between points of interest on the body of a musician playing an instrument and information about the instructor who teaches the musician to determine the points of interest based on the information about the instructor, The acquisition unit acquires image information representing the image of the area of ​​interest determined by the aforementioned determination unit, Includes, The information regarding the instructor includes cautionary information indicating performance guidelines provided by the instructor. Information processing system.

4. A determination unit that determines the points of interest based on the information about the instructor, using a table showing the correspondence between points of interest on the body of a performer playing an instrument and information about the instructor who instructs the performer, or using a trained model that has learned the relationship between the points of interest and the information about the instructor. The acquisition unit acquires image information representing the image of the area of ​​interest determined by the aforementioned determination unit, Includes, The type of instrument played by the aforementioned performer is the same as the type of instrument played by the aforementioned instructor. The information relating to the instructor includes information indicating the type of instrument played by the instructor. Information processing system.

5. A determination unit that determines the points of interest based on the information about the performer, using a table that shows the correspondence between information about the performer playing a musical instrument and points of interest on the performer's body. The acquisition unit acquires image information representing the image of the area of ​​interest determined by the aforementioned determination unit, Includes, The information relating to the performer includes the performer's identification information. Information processing system.

6. A determination unit that determines the points of interest based on the information about the performer, using a table that shows the correspondence between information about the performer playing a musical instrument and points of interest on the performer's body. The acquisition unit acquires image information representing the image of the area of ​​interest determined by the aforementioned determination unit, Includes, The information relating to the performer includes motion information indicating the performer's movements. Information processing system.

7. The determination unit further determines the point of interest based on instrument information indicating the type of instrument played by the performer. The information processing system according to any one of claims 1 to 6.

8. Further includes a identifying unit that identifies the instrument information using relevant information relating to the type of instrument played by the performer, The determination unit determines the area of ​​interest based on the instrument information identified by the identification unit. The information processing system according to claim 7.

9. The aforementioned related information is, Information indicating the sound output by the instrument played by the aforementioned performer, Information showing an image representing the instrument played by the aforementioned performer, Information indicating the musical score corresponding to the type of instrument played by the aforementioned performer, or This includes information indicating the combination of the type of instrument played by the aforementioned performer and the instruction schedule for that instrument. The information processing system according to claim 8.

10. The determination unit further determines the point of interest based on sound information indicating the sound output from the instrument of the type indicated by the instrument information, and the instrument information. The information processing system according to any one of claims 7 to 9.

11. The determination unit further determines the area of ​​interest based on sound information indicating the sound output from the instrument played by the performer. An information processing system according to any one of claims 1 to 9.

12. The determination unit further determines the point of interest based on the relationship between the musical score information representing the sound and the sound information. The information processing system according to claim 10 or 11.

13. The aforementioned determination unit further determines the points of interest based on cautionary information indicating notes regarding the performance. The information processing system according to claim 5 or 6.

14. A method of information processing performed by a computer, Using a table that shows the correspondence between points of interest on the body of a musician playing an instrument and information about the instructor who teaches the musician, the points of interest are determined based on the information about the instructor. Obtain image information representing the image of the determined area of ​​interest, The information relating to the instructor includes identification information of the instructor, movement information indicating the instructor's movements, or cautionary information indicating performance instructions given by the instructor. Information processing methods.

15. A method of information processing performed by a computer, Using a table showing the correspondence between points of interest on the performer's body while playing an instrument and information about the instructor who teaches the performer, or using a trained model that has learned the relationship between the points of interest and the instructor, the points of interest are determined based on the information about the instructor. Obtain image information representing the image of the determined area of ​​interest, The type of instrument played by the aforementioned performer is the same as the type of instrument played by the aforementioned instructor. The information relating to the instructor includes information indicating the type of instrument played by the instructor. Information processing methods.

16. A method of information processing performed by a computer, Using a table that shows the correspondence between information about a musician playing a musical instrument and points of interest on the musician's body, the points of interest are determined based on the information about the musician. Obtain image information representing the image of the determined area of ​​interest, The information relating to the performer includes identification information of the performer, or motion information indicating the performer's movements. Information processing methods.

17. On the computer, Using a table that shows the correspondence between points of interest on the body of a musician playing an instrument and information about the instructor who teaches the musician, the points of interest are determined based on the information about the instructor. Obtain image information representing the image of the determined area of ​​interest. Execute the process, The information relating to the instructor includes identification information of the instructor, movement information indicating the instructor's movements, or cautionary information indicating performance instructions given by the instructor. program.

18. On the computer, Using a table showing the correspondence between points of interest on the performer's body while playing an instrument and information about the instructor who teaches the performer, or using a trained model that has learned the relationship between the points of interest and the instructor, the points of interest are determined based on the information about the instructor. Obtain image information representing the image of the determined area of ​​interest. Execute the process, The type of instrument played by the aforementioned performer is the same as the type of instrument played by the aforementioned instructor. The information relating to the instructor includes information indicating the type of instrument played by the instructor. program.

19. On the computer, Using a table that shows the correspondence between information about a musician playing a musical instrument and points of interest on the musician's body, the points of interest are determined based on the information about the musician. Obtain image information representing the image of the determined area of ​​interest. Execute the process, The information relating to the performer includes identification information of the performer, or motion information indicating the performer's movements. program.

Images