METHOD FOR PROCESSING INFORMATION, INFORMATION PROCESSING SYSTEM AND STORAGE MEDIUM

Abstract

An information processing system 100 comprises an information acquirer 21 configured to acquire input information C, which includes finger information Y relating to a finger of a user playing a stringed instrument 200, an image of the fretboard of the stringed instrument 200, and sound information X about the sound of the stringed instrument 200 played by the user. An information generator 22 is configured to generate finger information Z indicating fingering by processing the acquired input information C using a generation model M that learns a relationship between training input information and training finger information.

Description

TECHNICAL FIELD

[0001] This revelation relates to a technique for analyzing the playing of a stringed instrument. STATE OF THE ART

[0002] Various techniques have been proposed to assist users in playing stringed instruments. For example, Patent Document 1 discloses a technique for displaying an image representing the fingering of a chord on a stringed instrument on a display. Related technology documentsPatent document

[0003] Patent Document 1: Japanese Patent Application Laid-Open No. 2005-241877 SUMMARY OF THE INVENTION Problems to be solved by the invention

[0004] When playing a stringed instrument, different fingerings can be used to produce the same note on the same instrument. When practicing a stringed instrument, a user may want to compare their fingering to a model fingering or the fingering of a specific player. Furthermore, the user may want to verify their fingering while playing the stringed instrument. Given these circumstances, one aspect of this disclosure is to provide fingering information to the user playing the stringed instrument. Means to solve the problems

[0005] To achieve the above-mentioned object, an information processing method according to one aspect of this disclosure comprises: (i) acquiring input information including: (ia) finger information related to the fingers of a user playing a stringed instrument and an image of a fingerboard of the stringed instrument; and (ib) sound information of the stringed instrument played by the user; and (ii) generating fingering information indicating the fingering style by processing the acquired input information using a generation model that learns a relationship between training input information and training fingering information.

[0006] An information processing system according to one aspect of this disclosure comprises: (i) an information collector configured to acquire input information including: (ia) fingering information related to the fingers of a user playing a stringed instrument and an image of a fingerboard of the stringed instrument; and (ib) sound information of the stringed instrument played by the user; and (ii) an information generator configured to generate fingering information indicating the fingering style by processing the acquired input information using a generation model that learns a relationship between training input information and training fingering information.

[0007] A program according to an aspect of this disclosure, causing a computer system to operate as follows: (i) an information collector configured to acquire input information including: (ia) fingering information related to the fingers of a user playing a stringed instrument and an image of a fingerboard of the stringed instrument; and (ib) sound information of the stringed instrument played by the user; and (ii) an information generator configured to generate fingering information indicative of the fingering style by processing the acquired input information using a generation model that learns a relationship between training input information and training fingering information. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 is a block diagram showing an example of the configuration of an information processing system. Fig. Figure 2 is a schematic diagram of a picture showing playing. Fig. Figure 3 is a block diagram showing an example of a functional configuration of the information processing system. Fig. Figure 4 is a flowchart showing image analysis procedures. Fig. Figure 5 is a schematic diagram of a reference image. Fig. Figure 6 is a flowchart showing game analysis procedures. Fig. Figure 7 is a block diagram showing an example configuration of a machine learning system. Fig. Figure 8 is a block diagram showing an example of the functional configuration of the machine learning system. Fig. Figure 9 is a flowchart showing machine learning techniques. Fig. 10 is a block diagram showing an example of the functional configuration of an information processing system according to a third embodiment. Fig. 11 is a block diagram showing an example of the functional configuration of an information processing system according to a fourth embodiment. Fig. 12 is a block diagram showing an example of the functional configuration of a machine learning system according to the fourth embodiment. Fig. 13 is a schematic diagram of a reference image according to a modification. Fig. 14 is a block diagram showing an example of the functional configuration of an information processing system according to a modification. Fig. 15 is a block diagram showing an example of the functional configuration of an information processing system according to a modification. EMBODIMENTS FOR CARRYING OUT THE INVENTION A: First Embodiment

[0008] Fig. 1 is a block diagram showing a configuration example of an information processing system 100 according to a first embodiment. The information processing system 100 is a computer system (performance analysis system) that analyzes the playing of a stringed instrument 200 by a user U. The stringed instrument 200 is, for example, a natural instrument such as an acoustic guitar that has a fingerboard and strings. The information processing system 100 according to the first embodiment analyzes the fingering of the user U when playing the stringed instrument 200. The term "fingering" refers to how the user U uses his / her fingers when playing the stringed instrument 200.Specifically, a combination of fingers that user U uses to fret strings on the fingerboard and the positions on the fingerboard where the user frets the strings are analyzed to determine the fingering style of user U when he / she plays the stringed instrument 200.

[0009] The information processing system 100 includes a controller 11, a storage medium 12, an operating device 13, a display 14, a sound receiver 15, and an image capture device 16. The information processing system 100 is a portable information device such as a smartphone or a tablet. Alternatively, the information processing system 100 is a portable or desktop information device, such as a personal computer. The information processing system 100 can be implemented by a single device or by more than one device.

[0010] The controller 11 includes one or more processors for controlling the operation of the information processing system 100. Specifically, the controller 11 includes one or more processors, such as CPUs (Central Processing Units), GPUs (Graphics Processing Units), SPUs (Sound Processing Units), DSPs (Digital Signal Processors), FPGAs (Field Programmable Gate Arrays), or ASICs (Application Specific Integrated Circuits).

[0011] The storage medium 12 includes one or more memories for storing programs executed by the controller 11 and a variety of data types used by the controller 11. The storage medium 12 may be a known recording medium, such as a semiconductor recording medium or a magnetic recording medium, or a combination of different types of recording media. For example, the storage medium 12 may be a portable recording medium that is connected to or disconnected from the information handling system 100, or a recording medium (e.g., cloud storage) that is accessed by the controller 11 over a network.

[0012] The operating device 13 is an input device that receives input operations from the user U. For example, the operating device 13 may be an input operator used by the user U or a touch panel for recognizing touch inputs from the user U. The display 14 displays a variety of images under the control of the controller 11. The display 14 may be one of various display panels, such as a liquid crystal display panel or an organic EL panel. The operating device 13 or the display 14, which is separate from the information processing system 100, may be connected to the information processing system 100 by cable or wirelessly.

[0013] The sound receiver 15 is a microphone that receives musical sounds produced by the stringed instrument 200 when played by the user U to generate an audio signal Qx. The audio signal Qx indicates a waveform of the musical sound produced by the stringed instrument 200. The sound receiver 15 is separate from the information processing system 100 and can be connected to the information processing system 100 either by cable or wirelessly. The illustration of an A / D converter for converting the audio signal Qx from analog to digital format is omitted for clarity.

[0014] The image capture device 16 captures images of the user U playing the stringed instrument 200 to generate an image signal Qy. The image signal Qy is a video signal representing the user U playing the stringed instrument 200. The image capture device 16 includes an optical system (e.g., a lens), an image sensor that receives the incident light from the optical system, and a processing circuit for generating an image signal Qy based on the amount of light received by the image sensor. The image capture device 16 is separate from the information processing system 100 and can be connected to the information processing system 100 either by cable or wirelessly.

[0015] Fig. 2 is an explanatory diagram of an image captured by the image pickup device 16. An image G represented by the image signal Qy (hereinafter referred to as a "performance image") includes a player image Ga and an instrument image Gb. The player image Ga represents the user U playing the stringed instrument 200. The instrument image Gb represents the stringed instrument 200 played by the user U. The player image Ga includes an image Ga1 representing the left hand of the user U (hereinafter referred to as a "left-hand image") and an image Ga2 representing the right hand of the user U (hereinafter referred to as a "right-hand image"). In the following description, it is assumed that the left hand of the user U is used to fret strings and the right hand is used to pluck strings. However, it is also possible that the left hand of the user U is used to pluck and the right hand is used to fret strings.The instrument image Gb includes an image Gb1 representing the fingerboard of the stringed instrument (hereinafter referred to as “fingerboard image”).

[0016] Fig. Figure 3 is a block diagram showing an example of a functional configuration of the information processing system 100. The controller 11 executes a program stored in the storage medium 12 to implement functions for analyzing the playing of the stringed instrument 200 by the user U (an information collector 21, an information generator 22, and a presentation processor 23).

[0017] The information collector 21 acquires input information C. The input information C is control data including sound information X and fingering information Y. The sound information X is data about the musical sound of the stringed instrument 200 played by the user U. The fingering information Y is data about a performance image G representing the user U playing the stringed instrument 200. The generation of the input information C by the information collector 21 is repeated in conjunction with the playing of the stringed instrument 200 by the user U. The information collector 21 according to the first embodiment includes an audio analyzer 211 and an image analyzer 212.

[0018] The audio analyzer 211 analyzes an audio signal Qx to generate sound information X. The sound information X according to the first embodiment identifies a pitch of a sound of the stringed instrument 200 played by the user U. Thus, the audio analyzer 211 estimates a pitch of the sound indicated by the audio signal Qx and generates sound information X for identifying the pitch. A known arbitrary analysis technique can be used to estimate a pitch indicated by the audio signal Qx.

[0019] The audio analyzer 211 sequentially detects an "onset" by analyzing the audio signal Qx. The "onset" is a time at which a sound is generated by the stringed instrument 200. More specifically, the audio analyzer 211 sequentially analyzes the volume of the audio signal Qx within a predetermined cycle and detects an "onset," which is a time at which the volume exceeds a predetermined threshold. Since a sound of the stringed instrument 200 is generated by the user U plucking a string, the "onset" is a time at which a string of the stringed instrument 200 is plucked by the user U.

[0020] The audio analyzer 211 generates sound information X upon detection of an onset. The sound information X is generated for each onset of the stringed instrument 200. More specifically, the audio analyzer 211 analyzes a sample of the audio signal Qx after a predetermined time (e.g., 150 milliseconds) has elapsed after each onset to generate sound information X. The sound information X at each onset represents a pitch of the musical sound at the onset.

[0021] The image analyzer 212 generates fingering information Y by analyzing the image signal Qy. The fingering information Y according to the first embodiment represents a left-hand image Ga1 of the user U and a fingerboard image Gb1 of the stringed instrument 200. The image analyzer 212 generates the fingering information Y upon detection of an onset by the audio analyzer 211. The fingering information Y is generated for each onset of the stringed instrument 200. For example, the image analyzer 212 analyzes the performance image G included in the image signal Qy after a predetermined time (e.g., 150 milliseconds) has elapsed after each onset to generate fingering information Y. The fingering information Y at each onset represents the left-hand image Ga1 and the fingerboard image Gb1.

[0022] Fig. 4 is a flowchart showing the processes Sa3 for generating finger information Y by the image analyzer 212 (hereinafter referred to as "image analysis processes"). The image analysis processes Sa3 begin with the detection of an onset. After starting the image analysis processes Sa3, the image analyzer 212 performs image recognition (Sa31). Image recognition is processing for extracting a left-hand image Ga1 of the user U and a fingerboard image Gb1 of the stringed instrument 200 from the performance image G represented by the image signal Qy. For example, image recognition is implemented by object recognition using a statistical model such as a deep neural network.

[0023] The image analyzer 212 performs the image conversion (Sa32). The image conversion is an image processing for converting a game image G, as shown in Fig. 2, a fingerboard image Gb1 is converted into an image representing the fingerboard from a predetermined orientation and distance. For example, the image analyzer 212 converts the performance image G so that the fingerboard image Gb1 fits within a rectangular reference image Gref along a predetermined direction. The conversion of both the left-hand image Ga1 of the user U and the fingerboard image Gb1 is performed. The image conversion can be implemented by known image processing such as projection conversion, in which a conversion matrix generated from the fingerboard image Gb1 and the reference image Gref is applied to the performance image G. The image analyzer 212 generates finger information Y representing the performance image G to which the image conversion has been applied.

[0024] As described above, the sound information X and the fingering information Y are generated for each onset. Specifically, the information collector 21 generates input information C for each onset of the stringed instrument 200. A time series of pieces of input information C corresponding to a plurality of different onsets is generated.

[0025] The Fig. The information generator 22 shown in Figure 3 generates the fingering information Z from the input information C. The fingering information Z is any type of data representing the fingering during the playing of the stringed instrument 200. Specifically, the fingering information Z identifies at least one finger used to finger a string of the stringed instrument 200 and a position where the finger is fingered. For example, a fingered position is identified by a combination of a string of the stringed instrument 200 and a fret on the fingerboard.

[0026] As described above, the input information C is generated for each onset, and the fingering information Z is generated by the information generator 22 for each onset. Thus, a time series of fingering information Z is generated for the plurality of different onsets. The generated fingering information Z for each onset represents a fingering to that onset. As is clear from the above description, in the first embodiment, the acquisition of the input information C and the generation of the fingering information Z are performed for each onset of the stringed instrument 200. This can prevent the generation of unnecessary fingering information when a string is pressed but not plucked. It should be noted that the acquisition of the input information C and the generation of the fingering information Z may be repeated for different, predetermined cycles of onsets.

[0027] A generation model M is used by the information generator 22 to generate the finger information Z. Specifically, the information generator 22 causes the generation model M to process the input information C to generate the finger information Z. The generation model M is a trained model that learns the relationships between the input information C and the finger information Z through machine learning. The generation model M provides statistically plausible finger information Z for the input information C.

[0028] The generation model M is implemented by (i) a program executed by the controller 11 that performs an operation to generate the finger information Z from the input information C, and (ii) variables applied to the operation (e.g., weighted values ​​and biases). The program and variables for the generation model M are stored in the storage device 12. The variables of the generation model M are preset using machine learning.

[0029] The generation model M, for example, comprises a deep neural network. The generation model M can be any type of deep neural network, for example, a recurrent neural network (RNN) or a convolutional neural network (CNN). The generation model M can be a combination of several types of deep neural networks. The generation model M can include an additional element such as long short-term memory (LSTM) or attention.

[0030] The presentation processor 23 provides the finger information Z to the user U. In particular, the presentation processor 23 displays a reference image R1 on the display 14, as shown in Fig. 5. The reference image R1 contains musical notation images B (B1 and B2) for use by the user U when playing the stringed instrument 200. The musical notation B1 is a staff line showing notations corresponding to the fingering information Z. The musical notation B2 is a tablature showing fingering corresponding to the fingering information Z. The musical notation B2 is an image containing six horizontal lines representing the strings of the stringed instrument 200. In the musical notation B2, numbers corresponding to the strings are displayed to indicate, in a time series, the frets at which the strings should be fingered. The presentation processor 23 generates musical note information P using the time series of the fingering information Z. The musical note information P can be data in any format that supports the Fig. 5. The presentation processor 23 displays the musical note images B represented by the musical note information P on the display 14.

[0031] Fig. 6 is a flowchart showing the processes Sa executed by the controller 11 (hereinafter referred to as "game analysis processes"). For example, the game analysis processes Sa begin when the operating device 13 receives an instruction from the user U.

[0032] After starting the game analysis process Sa, the controller 11 (the audio analyzer 211) waits until an audio signal Qx is analyzed and an onset is detected (Sa1: NO). If an onset is detected (Sa1: YES), the controller 11 (the audio analyzer 211) analyzes the audio signal Qx to generate sound information X (Sa2). In addition, the controller 11 (the image analyzer 212) performs the Fig. 4 to generate finger information Y. The order of generating the sound information X (Sa2) and the finger information Y (Sa3) can be reversed. Thus, input information C is generated for each onset of the stringed instrument 200. The input information C can be generated for a predetermined cycle.

[0033] The controller 11 (the information generator 22) causes the generation model M to process the input information C and generate finger information Z (Sa4). The controller 11 (the presentation processor 23) provides the finger information Z to the user U (Sa5 and Sa6). Specifically, the controller 11 generates musical note information P for the musical notation B from the finger information Z (Sa5) and displays the musical notation B on the display 14 (Sa6).

[0034] The controller 11 determines whether a predetermined stop condition is met (Sa7). The stop condition is, for example, a condition under which the controller 11 receives an instruction from the operating device 13 operated by the user U to terminate the performance analysis processes Sa. Alternatively, the stop condition may be a condition under which a predetermined time has elapsed since the last onset of the stringed instrument 200. If the stop condition is not met (Sa7: NO), the controller 11 proceeds to step Sa1. Thus, the acquisition of the input information C (Sa2 and Sa3), the generation of the fingering information Z (Sa4), and the display of the fingering information Z (Sa5 and Sa6) are repeated for each onset of the stringed instrument 200. In contrast, the performance analysis processes Sa end when the stop condition is met (Sa7: YES).

[0035] As is clear from the above description, in the first embodiment, the input information C including the sound information X and the fingering information Y is processed by the generation model M to generate the fingering information Z. Thereby, it is possible to generate fingering information Z for the following case: a musical sound (audio signal Qx) generated by the stringed instrument 200 during performance by the user U, and an image (image signal Qy) representing the user U's performance of the stringed instrument 200. In other words, it is possible to provide the fingering information Z for the user U's performance of the stringed instrument 200. Specifically, in the first embodiment, the musical score information P is generated using the fingering information Z. The display of the score images B enables the user U to effectively utilize the fingering information Z.

[0036] Fig. 7 is a block diagram showing an example of a configuration of a machine learning system 400 according to the first embodiment. The machine learning system 400 is a computer system in which a generation model M for use by the information processing system 100 is established using machine learning. The machine learning system 400 includes a controller 41 and a storage device 42.

[0037] The controller 41 includes one or more processors for controlling each element of the machine learning system 400. The controller 41 may include, for example, one or more processors, such as a CPU, a GPU, an SPU, a DSP, an FPGA, or an ASIC.

[0038] The storage device 42 includes one or more memories for storing a program executed by the controller 41 and a variety of data used by the controller 41. The storage device 42 may be a known recording medium, such as a magnetic recording medium or a semiconductor recording medium. The storage device 42 may include combinations of recording media types. The storage device 42 may be a portable recording medium that is connected to or disconnected from the machine learning system 400, or a recording medium (e.g., cloud storage) that the controller 41 accesses over a network.

[0039] Fig. 8 is a block diagram showing an example of a functional configuration of the machine learning system 400. The storage device 42 stores a plurality of training data T. Each of the plurality of training data T is teaching data that includes training input information Ct and training finger information Zt.

[0040] The training input information Ct includes sound information Xt and finger information Yt. The sound information Xt is data about the musical sound of the stringed instrument 201 played by a large number of players (hereinafter referred to as "reference players"). Specifically, the sound information Xt identifies a pitch of a tone produced by the stringed instrument 201 played by a reference player. The finger information Yt is data about a captured image of the left hand of the reference player and the fingerboard of the stringed instrument 201. The finger information Yt thus represents an image of the left hand of the reference player and an image of the fingerboard of the stringed instrument 201.

[0041] The fingering information Zt contained in the training data T is data representing the fingering of a reference player when playing the stringed instrument 201. In other words, the fingering information Zt contained in the training data T indicates a ground truth generated by the generation model M and used as input information Ct of the training data T.

[0042] Specifically, the finger information Zt identifies a finger number of the left hand of the reference player used to fret a string of the stringed instrument 201 and a position at which the string is fretted. The position at which the string is fretted is indicated by the finger information Zt and detected by a detector 250 installed in the stringed instrument 201. The detector 250 is an optical or mechanical sensor mounted, for example, on the fingerboard of the stringed instrument 201. A known technique, as disclosed in U.S. Patent No. 9,646,591, can be used to detect a position at which a string should be fretted, indicated by the finger information Zt.As is clear from the above description, the training finger information Zt is generated from a result provided by the detector 250 installed in the stringed instrument 201 that detects the playing of the reference player. This can reduce the time and effort required to prepare the training data T for use in the machine learning of the generation model M.

[0043] The controller 41 of the machine learning system 400 executes a program stored in the storage device 42 to implement functions for generating the generation model M (a training data acquirer 51 and a learning processor 52). The training data acquirer 51 acquires a plurality of training data T. The learning processor 52 establishes a generation model M through machine learning using the plurality of training data T.

[0044] Fig. 9 is a flowchart illustrating the methods Sb used by the controller 41 to create a generation model M through machine learning (hereinafter referred to as "machine learning methods"). The machine learning methods Sb begin, for example, upon receiving an instruction from a human operator of the machine learning system 400.

[0045] After starting the machine learning processes Sb, the controller 41 (the training data acquirer 51) selects one or more of the plurality of training data T (hereinafter referred to as "selected training data T") (Sb1). The controller 41 (the learning processor 52) repeatedly updates the coefficients of an initial or preliminary generation model M (hereinafter referred to as "preliminary model M0") using the selected training data T (Sb2 to Sb4).

[0046] The controller 41 causes the preliminary model M0 to process the input information Ct of the selected training data T to generate the finger information Z (Sb2). The controller 41 calculates a loss function representing an error between the finger information Z generated by the preliminary model M0 and the finger information Zt of the selected training data T (Sb3). The controller 41 updates the variables of the preliminary model M0 to reduce (ideally minimize) the loss function (Sb4). For example, an error backpropagation method is used to update the variables according to the loss function.

[0047] The controller 41 determines whether a predetermined stopping condition is met (Sb5). The stopping condition is met when the loss function is below a predetermined threshold or the change in the loss function is below a predetermined threshold. If the stopping condition is not met (Sb5: NO), the controller 41 selects training data T that has not been selected yet as the new selected training data T (Sb1). Until the stopping condition is met (Sb5: YES), the updating of the variables of the preliminary model M0 (Sb1 to Sb4) is repeated. If the stopping condition is met (Sb5: YES), the controller 41 terminates the machine learning processes Sb. The preliminary model M0 provided at the time the stopping condition is met is defined as the trained generation model M.

[0048] As is clear from the above description, the generation model M learns a potential relationship between the input information Ct and the finger information Zt contained in each of the plurality of training data T. The trained generation model M thus provides statistically plausible finger information Z for unknown input information C in accordance with the relationship.

[0049] The controller 41 transmits the generation model M established by the machine learning methods Sb to the information processing system 100. In particular, variables defining the generation model M are transmitted to the information processing system 100. The controller 11 of the information processing system 100 receives the generation model M from the machine learning system 400 and stores the generation model M in the storage device 12. B: Second embodiment

[0050] A second embodiment will be described below. In the embodiments described below, the same reference numerals are used for elements that have the same functions or effects as those described in the first embodiment. Detailed explanations of such elements will be omitted where appropriate.

[0051] The configuration and operation of an information processing system 100 in the second embodiment are the same as those in the first embodiment. The second embodiment provides the same effects as the first embodiment. In the second embodiment, the finger instruction Zt included in a training data set T and used in the machine learning methods Sb differs from the first embodiment.

[0052] In the first embodiment, the training data T includes (i) input information Ct (sound information Xt and finger information Yt) for playing by each reference player and (ii) finger information Zt for playing by each reference player. The training data T is used in the machine learning procedures Sb for the generation model M. The input information Ct and finger information Zt contained in the training data T are used for playing by the same reference player.

[0053] In the second embodiment, the input information Ct of the training data T indicates information (sound information Xt and fingering information Yt) for playing by a large number of reference players, as in the first embodiment. The fingering information Zt included in the training data T indicates the fingering when played by a specific player (hereinafter referred to as the "target player"). The target player may be a musical artist who plays the stringed instrument 200 with characteristic fingering or a music teacher who plays the stringed instrument 200 with a model fingering. Thus, in the second embodiment, the input information Ct included in the training data T is used for playing by one player (i.e., the reference player). The fingering information Zt included in the training data T is used for playing by another player (i.e., the target player).

[0054] A captured image of a target player playing the stringed instrument is analyzed to prepare fingering information Zt for the target player, which is included in the training data T. For example, the fingering information Zt is generated from an image of live music or from a music video in which the target player performs. This applies the finger skills typical of the target player to the fingering information Zt. For example, the following features are applied to the fingering information Zt: strings tend to be fretted at a high frequency in a certain area of ​​the fingerboard of the stringed instrument, and strings tend to be fretted at a high frequency by a certain finger of the left hand.

[0055] As is clear from the above description, the generation model M according to the second embodiment is applied to the performance by the user U (the sound information Xt and the finger information Yt), and generates finger information Z to which a fingering tendency of the target player is applied. For example, assuming that the target player plays a piece of music similar to the user U, the finger information Z shows the fingering that the target player would use. When the user U checks the musical notation B displayed based on the finger information Z, the user U becomes aware that the target player would play the piece of music with a fingering similar to the user U.

[0056] According to the second embodiment, a target player, such as a music artist or a music teacher, can provide a good customer experience by making their fingering information Z available to a large number of users U. Users U can also have a customer experience by practicing a stringed instrument while referring to the fingering information Z of a desired target player. C: Third embodiment

[0057] Fig. 10 is a block diagram showing an example of a functional configuration of an information processing system 100 according to a third embodiment. In the third embodiment, multiple generation models M are selectively used for different target players. Each generation model M corresponds to the generation model M according to the second embodiment. A generation model M for a target player is a trained model that learns a relationship between training input information Ct and training finger information Zt indicating the grip of the target player.

[0058] Specifically, in the third embodiment, a plurality of training data T is prepared for each target player. The machine learning methods Sb use the plurality of training data T for a target player, and a generation model M is created for each target player by the machine learning methods Sb. The generation model M for a corresponding target player is used for the performance by the user U (sound information Xt and finger information Yt) and generates finger information Z to which a grasping tendency of the target player is applied.

[0059] The user U selects one of the target players by operating the operating device 13. The information generator 22 receives the target player selected by the user U. The information generator 22 generates fingering information Z by causing a generation model M to process the input information C (Sa4). At this time, the generation model M is provided for the target player selected by the user U from the generation models M. As a result, the fingering information Z generated by the generation model M indicates the fingering assumed by the selected target player, assuming that the same piece of music is played by the selected target player and the user U.

[0060] The third embodiment offers the same effects as those achieved in the second embodiment. Specifically, in the third embodiment, one of the generation models M is selectively used for different target players. This makes it possible to generate finger information Z in which a gripping tendency is applied specifically for a target player. D: Fourth embodiment

[0061] Fig. 11 is a block diagram showing an example of a functional configuration of an information processing system 100 according to a fourth embodiment. The input information C according to the fourth embodiment includes identification information D in addition to sound information X and finger information Y, similar to the first embodiment. The identification information D is a code string for identifying one of the target players.

[0062] As in the third embodiment, the user U selects one of the target players by operating the operating device 13. The information acquirer 21 generates identification information D for the target player selected by the user U. The information acquirer 21 thus generates input information C including sound information X, finger information Y, and the identification information D.

[0063] Fig. 12 is a block diagram showing an example of a functional configuration of a machine learning system 400 according to the fourth embodiment. In the fourth embodiment, similar to the third embodiment, training data T is prepared for each target player. The training data T for a target player includes training identification information Dt in addition to sound information Xt and fingering information Yt, similar to the first embodiment. The identification information Dt is a code sequence for identifying one of the target players. The training data T for the target player includes fingering information Zt indicating the fingering of the target player when playing the stringed instrument 200. Thus, a tendency in the target player's performance of the stringed instrument 200 is applied to the fingering information Zt for the target player.

[0064] In the third embodiment, the machine learning methods Sb use a plurality of training data T for a target player, and the machine learning methods generate a generation model M for each target player. In the fourth embodiment, the machine learning methods Sb use training data T for a player, and the machine learning methods Sb generate a single generation model M for different target players. In other words, the generation model M according to the fourth embodiment is a model that learns a relationship between the following points: (i) training input information Ct for each of the target players, the training input information Ct including identification information D for a corresponding target player, and (ii) Training finger information Zt indicating the grip of the corresponding target player.

[0065] The generation model M is used for playing by the user U (sound information Xt and finger information Yt) and generates finger information Z in which a grasping tendency of the target player selected by the user U is applied.

[0066] As described above, the fourth embodiment provides the same effects as those achieved in the second embodiment. Specifically, the input information C in the fourth embodiment includes the identification information D for a corresponding target player. As a result, as in the third embodiment, it is possible to generate finger information Z in which a grasping tendency is applied specifically for a target player. E: Fifth embodiment

[0067] The presentation processor 23 according to the fifth embodiment displays on the display 14 a reference image R2 which is Fig. 13 by using finger information Z. A configuration and operation different from the presentation processor 23 are the same in the first to fourth embodiments. Therefore, the fifth embodiment provides the same effects as the first to fourth embodiments.

[0068] The reference image R2 includes virtual objects O in a virtual space. Each of the virtual objects O is a three-dimensional image showing a virtual player Oa playing a virtual stringed instrument Ob. The virtual player Oa has a left hand Oa1 gripping the stringed instrument Ob and a right hand Oa2 plucking the stringed instrument Ob. A state of the virtual objects O (specifically, the left hand Oa1) changes over time based on the finger information Z sequentially generated by the information generator 22. As described above, according to the fifth embodiment, the presentation processor 23 displays, on the display 14, the reference image R2 representing the virtual player Oa (Oa1 and Oa2) and the virtual stringed instrument Ob.

[0069] The fifth embodiment provides the same effects as those achieved in the first to fourth embodiments. Specifically, in the fifth embodiment, the virtual player Oa with the fingering indicated by the finger information Z is displayed on the display 14 along with the virtual stringed instrument Ob. This allows the user U to easily visually verify the fingering indicated by the finger information Z.

[0070] The display 14 can be mounted on an HMD (Head Mounted Display) worn on the head of the user U. The presentation processor 23 displays on the display 14 the virtual objects O (the player Oa and the stringed instrument Ob) captured by a virtual camera in the virtual space. The virtual objects O are displayed as a reference image R2. The presentation processor 23 dynamically controls the position and orientation of the virtual camera in the virtual space based on a movement (e.g., a position and orientation) of the head of the user U. This allows the user U to view the virtual objects O from any position and direction in the virtual space by moving their head accordingly. The HMD with the display 14 can be of the transparent type, in which the physical background behind the virtual objects O is visible to the user U.Alternatively, the HMD can be of the non-transparent type, in which the virtual objects O are displayed together with the background image of the virtual space. The transparent HMD displays the virtual objects O using, for example, augmented reality (AR) or mixed reality (MR). The non-transparent HMD displays the virtual objects O using, for example, virtual reality (VR).

[0071] The display 14 may be mounted on a terminal device that can communicate with the information processing system 100 via a network, such as the Internet. The presentation processor 23 transmits image data indicating the reference image R2 to the terminal device to display the reference image R2 on the display 14 of the terminal device. The display 14 of the terminal device may or may not be mounted on the head of the user U. Q: Modifications

[0072] Specific modifications applicable to each of the aspects described above are set forth below. Modes freely selected from the preceding embodiments and the following modifications can be combined with each other as desired, as long as such combination does not cause contradiction.

[0073] (1) In the foregoing embodiments, an exemplary aspect is given for displaying note images B indicated by finger information Z on the display 14. However, the application of the finger information Z is not limited to such an example. For example, as shown in Fig. 14, the presentation processor 23 can generate content N based on the fingering information Z and the sound information X. The content N includes the musical notation B generated from a time series of the fingering information Z and a time series of pitches, each of which is indicated by sound information X for a corresponding onset. When the content is played back by a playback device, the sound is reproduced at pitches indicated by the respective sound information X in conjunction with the display of the musical notation B. This allows a person viewing the content to hear the sound of the musical piece while viewing the musical notation B. The content is useful as teaching materials for practicing or instructing how to play the stringed instrument 200.

[0074] (2) In the foregoing embodiments, an exemplary aspect is described in which a pitch is identified by sound information X. However, the information identified by the sound information X is not limited to a pitch. For example, the sound information X may be a frequency response of an audio signal Qx. The frequency response of the audio signal Qx may include intensity spectra (amplitude spectra or power spectra) or Mel-Frequency Cepstrum Coefficients (MFCC). The sound information X may include a time series of samples comprising the audio signal Qx. As is clear from such examples, the sound information X is comprehensively represented as information about a sound of the performance of the stringed instrument 200 by the user U.

[0075] (3) In the foregoing embodiments, an exemplary aspect is given in which an audio signal Qx is analyzed to generate sound information X. However, the method for generating the sound information X is not limited to such embodiments. For example, as shown in Fig.15, the audio analyzer 211 generates sound information X based on the performance information E sequentially supplied from an electric stringed instrument 202. The electric stringed instrument 202 is a MIDI (Musical Instrument Digital Interface) instrument that outputs performance information E indicating performance by the user U. The performance information E is event data for identifying a pitch and intensity of a sound produced by the user U's performance. The performance information E is output from the electric stringed instrument 202 each time the user U plucks a string. For example, the audio analyzer 211 generates a pitch included in the performance information E, and the generated pitch is indicated by the sound information X. The audio analyzer 211 can detect an onset based on the performance information E.For example, the audio analyzer 211 detects a timing at which the performance information E for sound generation is supplied from the electric stringed instrument 202, and the detected timing is an onset.

[0076] (4) In the foregoing embodiments, an audio signal Qx is analyzed, and an onset of the stringed instrument 200 is thereby detected. However, the method for detecting an onset is not limited to such an example. For example, the image analyzer 212 may analyze an image signal Qy to detect a sound source of the stringed instrument 200. As described above, a player image Ga represented by the image signal Qy includes a right hand image Ga2 showing the right hand of the user U plucking strings. The image analyzer 212 extracts a right hand image Ga2 from the player image G and analyzes the change in the right hand image Ga2 to detect the plucking of a string. A timing at which the string is plucked by the user U is detected as an onset.

[0077] (5) Examples of techniques for playing the stringed instrument 200 (e.g., a guitar) include an arpeggio, where the notes are played one after the other, and a strum, where the notes forming a chord are played substantially simultaneously. In analyzing the performance of the stringed instrument 200 (particularly the onsets), the arpeggio can be distinguished from the strum. For example, an onset for a note (in the arpeggio) is detected when the notes are played one after the other at intervals that exceed a predetermined threshold. In contrast, the same onset for notes (strum) is detected when the notes are played at intervals that are below the threshold. Thus, the technique for playing the stringed instrument 200 can be used to detect onsets. Onsets can be represented discretely on a timeline.In one aspect where onsets are represented discretely, an onset is identified for notes played at intervals below the threshold.

[0078] (6) In the foregoing embodiments, an exemplary aspect of the finger information Y is given, which includes a left-hand image Ga1 and a fingerboard image Gb1. However, the finger information Y may also include a right-hand image Ga2 in addition to the left-hand image Ga1 and the fingerboard image Gb1. According to this configuration, it is possible to generate the finger information Z as the right-hand fretwork and left-hand plucking of the user U. Similarly, finger information Yt included in the input information Ct of the training data T may include a right-hand image used for plucking the strings with the right hand of a reference player.

[0079] (7) In the foregoing embodiments, an exemplary aspect of the finger information Y is given, which includes a player image Ga (a left-hand image Ga1 and a right-hand image Ga2) and an instrument image Gb (a fretboard image Gb1). However, the format of the finger information Y is arbitrary. The image analyzer 212 may generate, as finger information Y, the coordinates of each characteristic point extracted from a player image G. For example, the finger information Y may include the coordinates of each node (e.g., knuckles or fingertips) included in the left-hand image Ga1 of the user U. Alternatively, the finger information Y may indicate the coordinates of each point where a string and a fret intersect on the fretboard image Gb1 of the stringed instrument 200.In one aspect of a right hand image Ga2 reflected in the finger information Y, the finger information Y may include the coordinates of each node (e.g., knuckles or fingertips) included in a right hand image Ga2 of the user U. As is clear from such examples, the finger information Y is comprehensively represented as information about the player image Ga and the instrument image Gb.

[0080] (8) In the third embodiment, one of the generation models M can be selected according to an instruction from the user U. However, the method for selecting a generation model M is not limited to such an example. The method for selecting one of the target players is arbitrary. For example, the information generator 22 can select one of the generation models M according to an instruction from an external device or as a result of a predetermined numerical calculation. Likewise, in the fourth embodiment, the method for selecting one of the target players is arbitrary. For example, the information collector 21 can generate identification information D for one of the target players according to instructions from an external device or as a result of a predetermined numerical calculation.

[0081] (9) In the foregoing embodiments, an example of a deep neural network is given as the generation model M for generating finger information Z. However, the generation model M is not limited to such an example. For example, a statistical model such as an HMM (Hidden Markov Model) or an SVM (Support Vector Machine) may be used as the generation model M.

[0082] (10) In each of the foregoing embodiments, the generation model M is used to learn a relationship between the input information C and the finger information Z. However, the configuration and method for generating the finger information Z from the input information C are not limited to such an example. For example, a reference table in which the finger information Z is linked to the input information C may be used by the information generator 22 to generate the finger information Z. The reference table is a data table containing correspondences between the input information C and the finger information Z. The reference table is stored, for example, in the storage device 12. The information generator 22 searches the reference table for the finger information Z associated with the input information C acquired by the information collector 21.

[0083] (11) In the preceding embodiments, a generation model M is created by the machine learning system 400. However, the functions for creating the generation model M (the training data collector 51 and the learning processor 52) may be implemented in the information processing system 100.

[0084] (12) In the foregoing embodiments, an example of finger information Z indicating a finger number and a position at which a string is fretted is given. However, the finger information Z is not limited to such an example. For example, the finger information Z may indicate various methods of playing stringed instruments used for musical expressions, in addition to the generally used finger grips defined by a finger number and a position at which a string is fretted. Examples of methods of playing stringed instruments indicated by the finger information Z include vibrato, slide, glissando, pull-off, hammering, and choking. Well-known estimation models of musical expressions are used to estimate the method used when playing a stringed instrument.

[0085] (13) The type of stringed instrument 200 is freely selectable. The stringed instrument 200 is broadly represented as an instrument equipped with strings that vibrate to produce sound. The stringed instrument 200 includes a plucked instrument and a bowed instrument. The plucked instrument corresponds to the stringed instrument 200 in which strings vibrate to produce sound. Examples of the plucked instrument are an acoustic guitar, an electric guitar, an acoustic bass, an electric bass, a ukulele, a banjo, a mandolin, a harp, and a shamisen. The bowed instrument is a stringed instrument that produces sound when the strings are bowed. Examples of the bowed instrument are a violin, a viola, a cello, and a double bass. This disclosure applies to all types of such stringed instruments for performance analysis.

[0086] (14) For example, the information processing system 100 can be implemented by a server device that communicates with a terminal device such as a smartphone or tablet. For example, the information collector 21 of the information processing system 100 receives an audio signal Qx (or performance information E) and an image signal Qy from the terminal device. The information collector 21 generates sound information X based on the audio signal Qx and generates fingering information Y based on the image signal Qy. The information generator 22 generates fingering information Z from the input information C including the sound information X and the fingering information Y. The presentation processor 23 generates musical note information P from the fingering information Z and transmits the musical note information P to the terminal device. The terminal device displays the musical note images B represented by the musical note information P on its display.

[0087] In a configuration in which the audio analyzer 211 and the image analyzer 212 are mounted on the terminal device, the information collector 21 receives the sound information X and the finger information Y from the terminal device. As is clear from the foregoing description, the information collector 21 corresponds to an element for generating the sound information X and the finger information Y. Alternatively, the information collector 21 corresponds to an element for receiving the sound information X and the finger information Y from another device, such as the terminal device. In other words, the "acquisition" of the sound information X and the finger information Y includes both generation and reception.

[0088] In a configuration in which the presentation processor 23 is mounted on the terminal device, the finger information Z generated by the information generator 22 is transmitted from the information processing system 100 to the terminal device. The presentation processor 23 displays the musical note information P generated from the finger information Z on the display. As is clear from the foregoing description, the presentation processor 23 may be omitted from the information processing system 100.

[0089] (15) The functions of the information processing system 100 according to the preceding embodiments are implemented through the cooperation between one or more processors comprising the controller 11 and a program stored in the storage device 12. The program can be provided for use in a computer by storing it on a computer-readable recording medium. Preferred examples of a recording medium include a non-volatile recording medium, an optical recording medium (optical disk), such as a CD-ROM. Other examples of the recording medium include any type of known recording medium, such as a semiconductor recording medium or a magnetic recording medium. Examples of the non-volatile recording medium include any recording medium that is not a volatile transmission signal. A volatile recording medium is not excluded.In a configuration where the program is distributed from a distribution device over a network, the recording medium used to store the program in the distribution device corresponds to the non-volatile recording medium. G: Appendices

[0090] The following configurations can be derived from the preceding embodiments.

[0091] A method for processing information according to one aspect (Aspect 1) of this disclosure comprises: (i) acquiring input information, including: (ia) finger information related to the fingers of a user playing a stringed instrument and an image of a fingerboard of the stringed instrument; and (ib) sound information about the sound of the stringed instrument played by the user; and (ii) generating finger information indicative of fingering by processing the acquired input information using a generation model that learns a relationship between training input information and training finger information. In this aspect, the finger information is generated by processing the input information, including the finger information and the sound information, using the trained generation model.In other words, the fingering information can be provided that relates to the fingering when the user plays the stringed instrument.

[0092] The "finger information" is any data format of an image of the user's fingers and an image of the fingerboard of the stringed instrument. For example, the finger information may be image information representing an image of the user's finger and an image of the fingerboard of the stringed instrument. Alternatively, the finger information may be analysis information generated from the image information. For example, the analysis information may specify the coordinates of each node (e.g., knuckles or fingertips) of any of the user's fingers, a line segment between the nodes, the fingerboard, or a fret on the fingerboard.

[0093] "Sound information" is any data format for the sound produced when the user plays the stringed instrument. For example, sound information indicates the characteristics of the note played by the user. The characteristics are identified, for example, by analyzing an audio signal indicating the vibrations of the stringed instrument's strings. For example, for a stringed instrument that outputs performance information in MIDI format, sound information is generated to identify a pitch of the performance information. A time series of samples of an audio signal can be used as sound information.

[0094] "Fingering information" is any data format that indicates the fingering used to play a stringed instrument. For example, fingering information may include a finger number, which indicates a finger that frets a string, and a fret position (a combination of a fret and a string).

[0095] The generation model is a trained model that learns relationships between input information and finger information through machine learning. A variety of training data is used for machine learning of the generation model. The variety of training data includes training input information and training finger information (ground truth). Examples of the generation model include a variety of statistical models such as a deep neural network (DNN), HMM, and SVM.

[0096] An example (aspect 2) according to aspect 1 additionally comprises the detection of an onset of the stringed instrument, in which the acquisition of the input information and the generation of the fingering information are carried out for each onset.

[0097] In this aspect, the acquisition of input information and the generation of fingering information are performed for each onset of the stringed instrument. This avoids the unnecessary generation of fingering information when a string is fretted by the user U without performing a sound action. The "sound action" is an action that causes the stringed instrument to produce a sound when a string is fretted. Specifically, the sound action is an action of plucking a string of a plucked stringed instrument or an action of bowing a string of a bowed instrument.

[0098] An example (aspect 3) according to aspect 1 or 2 additionally includes generating musical note information indicating musical notation for playing the stringed instrument by the user using the fingering information. In this aspect, musical note information is generated using the fingering information. The fingering information can be effectively utilized by the user when outputting musical notation (e.g., displayed or printed). The "musical notation" represented by the "musical note information" is, for example, a tablature notation indicating fingered string positions of the strings of the stringed instrument. However, the musical information may also indicate a score notation indicating the note pitches to be played.

[0099] An example (aspect 4) according to any one of aspects 1 to 3 additionally comprises displaying on a display a reference image representing: a virtual player with fingering indicated by the finger information and a virtual stringed instrument played with the fingers.

[0100] In this aspect, a virtual finger with the fingering indicated by the fingering information and the virtual stringed instrument are displayed on the screen. This allows the user to easily visually verify the fingering indicated by the fingering information.

[0101] In an example (aspect 5) according to any one of aspects 4, the display is worn on the user's head. Displaying the reference image includes displaying on the display a captured image depicting the virtual player and the virtual stringed instrument in a virtual space. The captured image is captured by a virtual camera in a position and orientation in the virtual space that is controlled based on a movement of the user's head and displayed as a reference image. According to this aspect, the user can view the virtual player and the virtual stringed instrument from a desired position and direction.

[0102] In an example (aspect 6) according to aspect 4 or 5, displaying the reference image includes transmitting image data displaying the reference image to a terminal device via a network to display the reference image on the terminal device's display. According to this aspect, even if the terminal device does not have a finger information generation function, the user of the terminal device can view the virtual player and the virtual stringed instrument according to the finger information.

[0103] An example (aspect 7) according to any one of aspects 1 to 6 additionally comprises generating content based on the sound information and the finger information.

[0104] According to this aspect, it is possible to generate content to verify the correspondence between sound information and fingering information. This content is useful for practice or instruction when playing a stringed instrument.

[0105] In an example (aspect 8) according to any one of aspects 1 to 7, the input information includes identification information for one of the plurality of players. The generation model is a model that learns a relationship between (i) training input information for each of the plurality of players, wherein the training input information includes identification information for a corresponding player, and (ii) training fingering information indicating the fingering of the corresponding player.

[0106] In this aspect, the input information includes the player's identification information. This makes it possible to generate finger information that applies a finger inclination typical for each player.

[0107] In an example (aspect 9) according to any one of aspects 1 to 7, generating the finger information comprises generating the finger information by processing the acquired input information using one of the various generation models for the respective different players. Each of the plurality of generation models is a model that learns a relationship between the training input information and the training finger information indicative of the fingering of a player corresponding to the generation model.

[0108] In this aspect, one of the models is selectively used for each different player. This makes it possible to generate finger information that applies a finger inclination that is typical for each player.

[0109] In an example (aspect 10) according to any one of aspects 1 to 9, the training fingering information is generated by using a result provided by a detector installed in the stringed instrument that detects a player's performance. In this aspect, the training fingering information is generated by using a result provided by the detector installed in the stringed instrument. This reduces the time and effort required to create training data for use in the machine learning of the generation model.

[0110] An information processing system according to one aspect (aspect 11) of this disclosure comprises: (i) an information collector configured to acquire input information including: (ia) finger information related to the fingers of a user playing a stringed instrument and an image of a fingerboard of the stringed instrument; and (ib) sound information of the stringed instrument played by the user; and (ii) an information generator configured to generate finger information indicative of fingering by processing the acquired input information using a generation model that learns a relationship between training input information and training finger information.

[0111] A program according to one aspect of this disclosure (Aspect 12) causing a computer system to act as: (i) an information collector configured to collect input information including: (ia) fingering information related to the fingers of a user playing a stringed instrument and an image of a fingerboard of the stringed instrument; and (ib) sound information of the stringed instrument played by the user; and (ii) an information generator configured to generate fingering information indicative of fingering by processing the collected input information using a generation model that learns a relationship between training input information and training fingering information. Description of reference symbols

[0112] 100... information processing system, 200, 201... stringed instrument, 202... electric stringed instrument, 250... detector, 11, 41... controller, 12, 42... storage device, 13... operating device, 14... display, 15... sound receiver, 16... image acquisition device, 21... information collector, 211... audio analyzer, 212... image analyzer, 22... information generator, 23... presentation processor, 400... machine learning system, 51... training data acquirer, 52... learning processor. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] JP 2005-241877

[0003] US 9646591

[0042]

Claims

[1] A computer-implemented method for processing information, comprising: Collect input information that includes: Finger information relating to the fingers of a user playing a stringed instrument and an image of a fingerboard of the stringed instrument; and Sound information about the sound of the string instrument played by the user; and Generating finger information indicative of fingering by processing the acquired input information using a generation model that learns a relationship between training input information and training finger information. [2] The information processing method according to claim 1, further comprising: detecting an onset of the stringed instrument, wherein the acquisition of the input information and the generation of the fingering information are performed for each onset. [3] The information processing method according to claim 1 or 2, further comprising: Generating musical note information using the fingering information indicating musical notation for playing the stringed instrument by the user. [4] The information processing method according to any one of claims 1 to 3, further comprising: displaying on a display a reference image representing: a virtual player with a fingering indicated by the finger information; and a virtual string instrument played with the fingers. [5] The information processing method according to claim 4, wherein: the display is worn on the user’s head, displaying the reference image comprises displaying on the display a captured image representing the virtual player and the virtual stringed instrument in a virtual space, and the captured image is captured by a virtual camera in a position and orientation in virtual space that is controlled based on a movement of the user's head and is displayed as a reference image. [6] The information processing method according to claim 4 or 5, wherein displaying the reference image comprises transmitting image data displaying the reference image to a terminal device via a network to display the reference image on the display of the terminal device. [7] The information processing method according to any one of claims 1 to 6, further comprising: generating contents based on the sound information and the finger information. [8] The information processing method according to any one of claims 1 to 7, wherein: the input information includes identification information for one of the plurality of players, and the generation model is a model that defines a relationship between: Training input information for each of the plurality of players, wherein the training input information includes identification information for a respective player, and Training finger information that indicates the fingering of the corresponding player is learned. [9] The information processing method according to any one of claims 1 to 7, wherein: generating the finger information comprises generating the finger information by processing the captured input information using one of the different generation models for different players, and each of the multiple generation models is a model that represents a relationship between: the training input information; and the training finger information indicating the fingering of a player corresponding to the generation model. [10] The information processing method according to any one of claims 1 to 9, wherein the training finger information is generated using a result provided by a detector installed in the stringed instrument that detects a player's performance. [11] An information processing system comprising: an information collector configured to collect input information comprising: Finger information relating to the fingers of a user playing a stringed instrument and an image of a fingerboard of the stringed instrument; and Sound information about the sound of the string instrument played by the user; and an information generator configured to generate finger information indicative of fingering by processing the acquired input information using a generation model that learns a relationship between training input information and training finger information. [12] A program that causes a computer system to act as follows: an information collector configured to collect input information that includes: Finger information relating to the fingers of a user playing a stringed instrument and an image of a fingerboard of the stringed instrument; and Sound information about the sound of the string instrument played by the user; and an information generator configured to generate finger information indicative of fingering by processing the acquired input information using a generation model that learns a relationship between training input information and training finger information.

Images