Information processing device, information processing method, and program

Abstract

The present invention comprises: a detection unit that detects, as a creation motion for creating a creation target, a first musical performance motion that is a motion of musical performance by a user, and detects a second musical performance motion that is a motion of musical performance by the user and that has occurred after the first musical performance motion; a generation unit that, on the basis of the creation motion and as musical performance information from which the creation target can be created, generates first musical performance information based on the first musical performance motion and second musical performance information in which the first musical performance information has been changed on the basis of the second musical performance motion; and an output unit that outputs the second musical performance information.

Description

Information processing device, information processing method, and program

[0001] The present invention relates to an information processing device, an information processing method, and a program.

[0002] There is a demand for finding new value in signal processing devices such as electronic musical instruments, for example, a demand for finding a "sense of collaboration" by creating a creative object such as a performance together with the device. Patent Document 1 discloses a technology that generates new music based on the music that a user listens to while exercising.

[0003] Japanese Patent Application Laid-Open No. 2009-237408

[0004] However, it would be more desirable to be able to create music based on more free input, for example, various actions from the user.

[0005] The present invention has been made in consideration of the above circumstances, and its purpose is to provide an information processing device, an information processing method, and a program that can output performance-related information in response to various user actions.

[0006] In order to solve the above-mentioned problems, one aspect of the present invention is an information processing device that includes a detection unit that detects creative actions that create a creative object, a generation unit that generates performance information that can create the creative object based on the creative actions, and an output unit that outputs the performance information.

[0007] Another aspect of the present invention is an information processing method performed by an information processing device that is a computer, which detects creative actions that create a creative object, generates performance information that can create the creative object based on the creative actions, and outputs the performance information.

[0008] Another aspect of the present invention is a program that causes an information processing device, which is a computer, to detect creative actions that create a creative object, generate performance information that can create the creative object based on the creative actions, and output the performance information.

[0009] According to the present invention, it is possible to output performance-related information in response to various user actions.

[0010] FIG. 1 is a diagram illustrating an overview of a creation support system according to an embodiment. FIG. 2 is a block diagram illustrating an example of the configuration of an information processing device 10 according to a first embodiment. FIG. 3 is a flowchart illustrating the flow of processing performed by the information processing device 10 according to the first embodiment. FIG. 4 is a flowchart illustrating the flow of processing performed by the information processing device 10 according to a second embodiment. FIG. 5 is a diagram illustrating the processing performed by the information processing device 10 according to a third embodiment. FIG. 6 is a diagram illustrating the processing performed by the information processing device 10 according to the third embodiment. FIG. 7 is a flowchart illustrating the processing performed by the information processing device 10 according to a fourth embodiment. FIG. 8 is a flowchart illustrating the processing performed by the information processing device 10 according to a modified example of the fourth embodiment. FIG. 9 is a flowchart illustrating the processing performed by the information processing device 10 according to a fifth embodiment. FIG. 10 is a diagram illustrating a modified example of an embodiment.

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] (Regarding Matters Common to the Embodiments) First, matters common to the following embodiments will be described. FIG. 1 is a diagram illustrating an overview of a creative support system according to an embodiment. FIG. 1 schematically illustrates an information processing device 10 as an instrument (electronic piano) and a user U playing the instrument. While the example in this diagram illustrates an example in which the information processing device 10 is an electronic instrument, the present invention is not limited to this. The information processing device 10 may be any device other than an electronic instrument, such as a smartphone, tablet, personal computer, cloud server, or physical server. Furthermore, each function of the information processing device 10 may be implemented by linking an instrument with a device, for example, linking an electronic instrument with a server.

[0013] The creation support system detects the movements of the user U, particularly the movements of creating a creation object (creative movements). The creative object here may be any, but may be, for example, a creative work created with the movements of the user U, such as a musical performance (including singing), a dance, or an acting performance. The information processing device 10 detects the creative movements of the user U and generates performance information based on the detected creative movements.

[0014] The performance information is information about a performance that can be used to create a creative subject. For example, when a user U is performing a creative action such as playing the piano, the performance information is information such as sounds that can provide suggestions for improving the performance. Each embodiment will be described below in order, and the relationship between the creative action and the performance information will be described in detail.

[0015] (Regarding the First Embodiment) First, the first embodiment will be described. In this embodiment, the information processing device 10 is an electronic musical instrument. The user U plays an instrument other than the electronic instrument of the information processing device 10, such as an acoustic instrument or another electronic instrument. The information processing device 10 outputs, as performance information, performance sounds (co-performing sounds) that accompany the performance movements of the user U.

[0016] The information processing device 10 may be integrated with the instrument played by the user U. In this case, the information processing device 10 outputs a performance sound played by the user U along with a corresponding sound that matches the performance sound. Alternatively, the information processing device 10 may be configured as a part of the instrument played by the user U. In this case, the information processing device 10 is an electronic drum used in addition to an acoustic drum.

[0017] 2 is a block diagram showing an example of the configuration of the information processing device 10 according to the first embodiment. As shown in FIG. 2, the information processing device 10 includes, for example, a detection unit 11, a generation unit 12, an output unit 13, and a storage unit 14. Note that the information processing device 10 may not include the storage unit 14. In this case, the information processing device 10 is configured to realize the functions performed by the information processing device 10 using the storage unit 14 by using a storage unit included in an external server or the like provided outside the information processing device 10 as the storage unit 14.

[0018] The detection unit 11 detects the playing motion. Any method may be used to detect the motion of the user U. For example, the playing motion can be detected by applying a technique such as motion capture to an image of the user U's motion captured by a camera or the like. Alternatively, the detection unit 11 may detect the image of the user U's motion itself as the playing motion. The detection unit 11 outputs the detected playing motion to the generation unit 12.

[0019] The generating unit 12 acquires the playing motions from the detecting unit 11 and generates playing information based on the acquired playing motions. A specific method for the generating unit 12 to generate playing information will be described in detail later.

[0020] The output unit 13 outputs the performance information generated by the generation unit 12. The output unit 13 outputs the performance information according to the type of the performance information. For example, if the performance information is sound information, the output unit 13 is configured to include a speaker and outputs sounds corresponding to the performance information through the speaker. For example, if the performance information is text information, the output unit 13 is configured to include a display and displays text corresponding to the performance information through the display.

[0021] The storage unit 14 is configured by a storage medium such as a hard disk drive (HDD), flash memory, electrically erasable programmable read-only memory (EEPROM), random access read / write memory (RAM), read-only memory (ROM), or a combination of these. The storage unit 14 stores programs for executing various processes in the information processing device 10 and temporary data used when performing the various processes.

[0022] The storage unit 14 stores, for example, performance action information 140, performance sound information 141, and performance information 142. The performance action information 140 is information indicating performance actions detected by the detection unit 11. The performance sound information 141 is information indicating performance sounds corresponding to the performance actions. The performance information 142 is performance information generated by the generation unit 12.

[0023] (Method by which the generation unit 12 generates performance information) For example, the generation unit 12 changes the settings of the sound output by the information processing device 10 as an electronic musical instrument, such as the tone color and volume, in accordance with the performance actions of the user U. For example, if the information processing device 10 is an electronic drum used in addition to an acoustic drum, the information processing device 10 outputs the sound of the electronic drum operated by the user U, and changes the tone color and volume of the electronic drum in accordance with the performance actions on the acoustic drum.

[0024] For example, the generation unit 12 determines whether to change the tone or volume depending on body movements, gaze, nods, facial expressions, gestures, hand movements, and other movements extracted from the performance movements of the user U. Any conventional method may be used to extract gaze and other movements from the performance movements of the user U. For example, gaze can be extracted by applying technology such as eye tracking to an image of the user's face. It is also possible to apply facial expression recognition technology that extracts facial feature points, such as the forehead, edge of the eyebrows, corners of the eyes, corners of the eyes, and corners of the mouth, from the image of the user's face and estimates the facial expression based on time-series changes in the extracted feature points.

[0025] The generation unit 12 stores in advance a setting table that associates body movements with setting values ​​such as tone colors. The generation unit 12 references the setting table according to the body movements extracted from the performance movements and acquires the setting values ​​associated with the setting table. The generation unit 12 changes the setting values ​​such as tone colors of the electronic musical instrument of the information processing device 10 according to the acquired setting values.

[0026] Furthermore, the generation unit 12 may determine whether the change is in line with the user's intention based on the user U's action extracted from the performance action after the change in tone, etc. For example, the generation unit 12 stores in advance an action table that associates OK actions with NG actions. For example, an OK action is a nodding action, and an NG action is a shaking of the head from side to side. The generation unit 12 extracts the user U's action from the performance action detected after the change in tone, etc., and if the extracted action is OK, maintains the change. On the other hand, if the extracted action is NG, the generation unit 12 cancels the change and returns to the original setting.

[0027] If the extracted motion is an inappropriate motion, the generation unit 12 may review the association between the body motion and the setting value used for the change in the setting table. In this case, the setting table sets a reliability for each combination of body motion and setting value. The reliability is the degree to which the setting is in line with the user's intention. If the user performs an inappropriate motion after changing the setting, the generation unit 12 changes the reliability for the combination of body motion and setting value corresponding to the change in the setting table to a smaller value. In this case, the generation unit 12 refers to the setting table according to the user's body motion, acquires the setting value associated with the body motion, and acquires the reliability associated with the setting value. The generation unit 12 determines whether to change the setting according to the acquired reliability. For example, the generation unit 12 determines to change the setting if the reliability is equal to or greater than a threshold, and determines not to change the setting if the reliability is less than the threshold.

[0028] Furthermore, in general, when a user wants to give instructions to a fellow performer, such as to play more energetically, quieter, faster, or slower, the user often performs some kind of movement (gesture). Even when conveying the same instruction, the same gesture is not necessarily used uniformly, and may vary depending on the performer, the song being played, the timing of the instruction, etc. It is desirable to be able to change the settings for such various gestures in accordance with the user U's intentions.

[0029] As a countermeasure to this, for example, the generation unit 12 may associate multiple setting values ​​with the same body movement in the setting table and associate a reliability level with each of the multiple setting values. For example, the generation unit 12 changes the setting value to one of the setting values ​​associated with a certain movement (gesture). Then, based on the user U's movement after the change (OK movement or NG movement), the generation unit 12 determines whether the change to the setting value was in line with the user's intention and reflects this in the reliability level for each setting value. Then, the generation unit 12 changes the setting value for a certain movement (gesture) to one with a higher reliability level. This makes it possible to change the setting value in line with the user's intention.

[0030] Furthermore, the performer may change the fingering used during performance depending on, for example, the performer's mood. The generation unit 12 may change the tone color and volume of the information processing device 10 in response to such changes in fingering. In this case, the detection unit 11 has a function for detecting performance movements (e.g., motion capture) and a function for collecting sound (e.g., a microphone). The detection unit 11 detects performance movements using motion capture or the like, acquires performance sounds corresponding to the performance movements using the microphone, and outputs the acquired performance movements and performance sounds to the generation unit 12. The generation unit 12 extracts fingerings from the performance movements acquired by the detection unit 11, associates the extracted fingerings with the performance sounds, and stores the extracted fingerings as performance information for each group of performance sounds (e.g., phrases). The generation unit 12 also references the performance information based on the performance sounds (phrases) acquired from the detection unit 11. If the same performance sounds (phrases) are stored, the generation unit 12 acquires the performance information. The generating unit 12 compares the fingering in the performance information with the fingering acquired this time from the detecting unit 11, and if the fingerings differ from each other, changes the tone color and volume.

[0031] Furthermore, in scenes where multiple people perform together, such as band or ensemble practice, there are cases where performance sounds are recorded or parts are practiced in timed intervals. In such cases, the task of turning on or off devices such as recorders or timers may arise. During a performance, the hands are often occupied, and it can be troublesome to go to the location where the devices are installed and operate them. The generation unit 12 may have a function to support such tedious tasks, specifically, operate devices corresponding to instructions based on the movements of a person giving instructions such as recording. In this case, the information processing device 10 has a recording function (e.g., a recorder) and a time measurement function (e.g., a timer). The generation unit 12 may also store in advance, for example, a task table that associates body movements with tasks. The generation unit 12 references the task table according to the body movements extracted from the performance movements and acquires the tasks associated with the task table. The generation unit 12 operates the devices according to the acquired tasks. When the information processing device 10 measures the time using a timer, it notifies the user that the measured time has been reached by sounding an alarm or the like.

[0032] The generation unit 12 may be configured to present the remaining time until the measurement time is reached during measurement by a timer in addition to the operation of a device. In this case, the information processing device 10 includes a speaker and a display. When a specific action for checking the remaining time is performed, the generation unit 12 announces the remaining time through the speaker or displays the remaining time on the display.

[0033] Furthermore, when multiple people are practicing together, it may be difficult for each person to properly communicate the signal when it is time to stop playing the part practice, which can result in a long wait before the next practice can begin. In such cases, the generation unit 12 may notify the user that the measurement time has expired by, for example, flashing a light. In this case, the information processing device 10 includes a display. The generation unit 12 notifies the user that the measurement time has expired by displaying a message on the display or flashing a light on the display.

[0034] Furthermore, when multiple people practice, it is expected that the information processing device 10 will be operated by multiple people. Even in such cases, it is desirable for the information processing device 10 to operate smoothly. To address this issue, for example, the detection unit 11 detects the performance movements of each of multiple people in a specific space, such as a practice studio. The generation unit 12 determines whether each performance movement includes a specific movement, and if the specific movement is included, executes an equipment operation corresponding to the movement. Then, if the performance movements detected after executing the equipment operation include an OK movement or an NG movement, the generation unit 12 sets a reliability based on the movement. If the person who performed the specific movement and the person who performed the OK movement or the NG movement are different people, the generation unit 12 may set a reliability based on the movement if the OK movement or the NG movement was performed within a certain time after the setting or equipment operation was executed. This makes it possible to set a reliability for an operation even if an operation performed based on an instruction from one person is confirmed or denied by another person.

[0035] Furthermore, when people try to play together, they often go through a process where they start out awkward but gradually develop a more synchronized performance. There is a demand for enjoying this process, where the performance is created through mutual cooperation, rather than a one-way process. The information processing device 10 of this embodiment may be configured to meet such demands.

[0036] In this case, the generation unit 12 generates performance information using, for example, a trained model. The trained model is a model trained using a training data set in which performance actions are associated with corresponding co-performed notes labeled (OK or NG). Here, the co-performed notes labeled OK are those that many users find favorable. On the other hand, the co-performed notes labeled NG are those that users find unfavorable. Whether a given note is favorable or unfavorable is determined, for example, by an experienced performer. By training using such a training data set, the trained model trains a correspondence between performance actions and co-performed notes that many users find favorable. By repeating the training, the trained model becomes able to accurately predict, for untrained performance actions, the co-performed notes that many users find favorable.

[0037] The generation unit 12 performs additional training using such a trained model. By performing the additional training, it becomes possible to predict co-performing sounds that are particularly tailored to the preferences of the user U. For example, the generation unit 12 inputs performance actions into the trained model. The trained model predicts and outputs co-performing sounds (co-performing sounds before the additional training) that many users will find preferable based on the input performance actions (performance actions before the additional training). The generation unit 12 uses the co-performing sounds output by the trained model as performance information. As a result, the co-performing sounds that many users will find preferable are output from the information processing device 10 via the output unit 13. The user U listens to the co-performing sounds output from the information processing device 10 and performs an OK action if they are preferable, or an NG action if they are not preferable. The information processing device 10 detects such OK actions or NG actions via the detection unit 11.

[0038] The generation unit 12 generates a training data set for additional training based on the OK or NG actions detected by the detection unit 11. The generation unit 12 assigns labels corresponding to OK or NG actions to the previously output co-performing sounds (co-performing sounds before the additional training). Specifically, if an OK action is performed on the co-performing sounds, the generation unit 12 assigns an OK label to the co-performing sounds, and if an NG action is performed on the co-performing sounds, the generation unit 12 assigns an NG label to the co-performing sounds. The generation unit 12 generates a training data set for additional training by associating the labeled co-performing sounds with performance actions (performance actions before the additional training) and stores the generated training data set for additional training. When the number of training data sets for additional training reaches a certain number, the generation unit 12 trains a trained model using the training data set for additional training. The generation unit 12 predicts co-performing sounds for performance actions using a trained model trained on the training data set for additional training. This makes it possible to output co-performing sounds that correspond to the preferences of the user U.

[0039] Furthermore, when two people perform together, there may be interactions in which they respond to each other and change their own performance in response to the other person's performance. The information processing device 10 according to the embodiment may be configured to realize such interactions.

[0040] For example, if the instrument played by the user U and the information processing device 10 are separate entities, the detection unit 11 first detects a playing action (first playing action) performed by the user U. The generation unit 12 generates playing information (first playing information) based on the first playing action. The output unit 13 outputs the first playing information. The detection unit then detects a playing action (second playing action) performed by the user U that occurred after the previous first playing action. The generation unit 12 generates information obtained by modifying the first playing information based on the second playing action as second playing information. The output unit 13 outputs the second playing information.

[0041] Alternatively, when the instrument played by the user U is integrated with the information processing device 10, the detection unit 11 first detects a playing action (first playing action) performed by the user U. The generation unit 12 generates playing information (first playing information) based on the first playing action. The output unit 13 outputs playing sounds (first playing sounds) corresponding to the first playing action and also outputs the first playing information. The detection unit then detects a playing action (second playing action) performed by the user U that occurred after the previous first playing action. The generation unit 12 generates second playing information by modifying the first playing information based on the second playing action. The output unit 13 outputs the second playing information when outputting second playing sounds corresponding to the second playing action.

[0042] For example, the generation unit 12 extracts features from the first and second performance actions and modifies the first performance information based on the difference between the extracted features. The extracted features may be any feature of the performance actions, such as the size of gestures. Furthermore, the difference in features may be, for example, the distance between a feature value corresponding to the first performance action and a feature value corresponding to the second performance action in the feature space of the performance actions. For example, when generating collaborative sounds as performance information, if the second performance action has larger gestures and movements than the first performance action, the generation unit 12 outputs, as the second performance information, collaborative sounds that are louder than the collaborative sounds (first performance information) corresponding to the first performance action. On the other hand, if the second performance action has smaller gestures and movements than the first performance action, the generation unit 12 outputs, as the second performance information, collaborative sounds that are quieter than the collaborative sounds (first performance information) corresponding to the first performance action.

[0043] The generating unit 12 may also determine whether to use information obtained by modifying the first performance information as the second performance information or to generate the second performance information from the second performance without using the first performance information, based on the difference in characteristics between the first performance action and the second performance action. If the difference in characteristics is large, it is considered that the user U was influenced by the first performance information and performed a second performance action with characteristics different from those of the first performance action. On the other hand, if the difference in characteristics is small, it is considered that the user U was hardly influenced by the first performance information and performed a second performance action with characteristics similar to those of the first performance action. If the user U is influenced by the first performance information, the generating unit 12 generates information obtained by modifying the first performance information as the second performance information. This allows information that is likely to influence the user U to be generated as the second performance information. On the other hand, if the user U is hardly influenced by the first performance information, the generating unit 12 generates the second performance information based on the second performance action without using the first performance information. This allows new second performance information to be generated, excluding information that does not affect the user U, and enables the output of a collaborative sound that influences the user U and elicits a response from the user U.

[0044] 3 is a flowchart showing the flow of processing performed by the information processing device 10 in this embodiment. First, the information processing device 10 determines whether a performance action has been detected (step S10). If a performance action has been detected, the information processing device 10 stores the currently detected performance action (second performance action) in the performance action information 140 (step S11). The information processing device 10 determines whether the difference between the features of the previously detected performance action (first performance action) and the features of the currently detected second performance action, for example, the distance in feature space, is equal to or greater than a threshold value (step S12).

[0045] If the difference is equal to or greater than the threshold, the information processing device 10 generates second performance information by modifying the first performance information based on the second performance motion (step S13). On the other hand, if the difference is less than the threshold, the information processing device 10 generates second performance information based only on the second performance motion without using the first performance information (step S14). The information processing device 10 stores the generated second performance information in the performance information 142 (step S15). The information processing device 10 outputs the second performance information (step S16).

[0046] As described above, the information processing device 10 in the embodiment includes a detection unit 11, a generation unit 12, and an output unit 13. The detection unit 11 detects creative actions that create a creative object. The generation unit 12 generates performance information that enables creation of a creative object based on the creative actions. The output unit 13 outputs the performance information. As a result, the information processing device 10 in the embodiment can output information that influences the creation, such as a performance, by the user U and can help create a creative object, such as information that provides hints for creating a creative work from the user U's actions, or information that stimulates the user U's creative motivation. As a result, performance-related output can be provided in response to various user actions.

[0047] Furthermore, in the information processing device 10 of the first embodiment, the detection unit 11 detects, as a creative action, a first performance action, which is an action of the user U performing a musical performance. The generation unit 12 generates first performance information based on the first performance action. The output unit 13 outputs the first performance information. The generation unit 12 also detects a second performance action, which is an action of the user U performing a musical performance that occurs after the first performance action. The generation unit 12 generates information by modifying the first performance information based on the second performance action as second performance information. The output unit 13 outputs the second performance information. As a result, the information processing device 10 of the embodiment can output information that further enhances the performance based on the performance action of the user U. Furthermore, information (second performance information) that further enhances the performance can be output in accordance with the user U's action (second performance action) influenced by the information (first performance information). This makes it possible to provide new value to the user U, as if they were performing together with a human, influencing each other to create a performance.

[0048] Furthermore, in the information processing device 10 of the first embodiment, the detection unit 11 detects, as creative actions, performance actions, which are actions of the user performing a musical instrument. The generation unit 12 generates, as performance information, sounds that are changed according to the body movements extracted from the performance actions. As a result, the information processing device 10 of the first embodiment can output sounds that respond to gestures made by the user U. Even in situations where it is difficult to perform operations or settings other than performance, such as when both hands of the user U are occupied while playing, the user U can change settings such as tone color and volume, or operate devices such as a recorder or timer.

[0049] (Second Embodiment) Here, a second embodiment will be described. In this embodiment, the information processing device 10 is an output device that outputs music generated using a PC (personal computer) or the like, so-called programmed music. For example, programmed music can be created by inputting information about the sounds to be performed in advance using a music sequencer or drum machine provided in music generation application software (music generation app) installed on the PC. The information processing device 10 includes a sound generation unit such as a sound source module or software synthesizer as an electronic musical instrument, and outputs sounds generated by the sound generation unit based on the information about the programmed music.

[0050] The generating unit 12 generates, as performance information, performance sounds corresponding to the performance actions and sounds obtained by changing the sounds to be output from pre-programmed music in accordance with the characteristics of the performance sounds.

[0051] For example, the generation unit 12 outputs programmed music such as EDM (electronic dance music), and, in accordance with the user U's performance actions, identifies the tone included in the programmed music that corresponds to the instrument being played by the user U, and replaces the identified tone with the sound being played by the user U (performance sound).

[0052] The generation unit 12 pre-stores, for example, an instrument table in which performance movements are associated with instrument information indicating the corresponding instrument. The generation unit 12 references the instrument table based on the performance movements and acquires the instrument information associated with the instrument table. Furthermore, the generation unit 12 determines, based on the acquired instrument information, whether the timbre of the instrument corresponding to the instrument information is used in the programmed music. If the timbre of the instrument corresponding to the instrument information is used in the programmed music, the generation unit 12 mutes the timbre of the instrument corresponding to the instrument information. This results in a music piece in which some parts of the programmed music are replaced with live performances by the user U. By replacing wind instrument parts such as a trumpet with live performances, EDM can be made more exciting. By practicing only some of the performance parts, the user U can perform an exciting EDM performance. Furthermore, by applying such a performance style, it is possible to increase the number of scenes in which real instruments are used.

[0053] The generation unit 12 may also change the sound output from the programmed music piece according to the characteristics of the performance sound corresponding to the performance movements of the user U. For example, the generation unit 12 determines the performance level based on the performance sound corresponding to the performance movements. The performance level is an indicator of the degree of proficiency in playing and can be classified, for example, into beginner, intermediate, and advanced. For example, the generation unit 12 pre-stores performance sounds for each music piece, each corresponding to a beginner, intermediate, and advanced performance level. Features are extracted from the performance sound by the user U. The extracted performance sound features may be any feature, such as a time series change in volume or frequency characteristics. The generation unit 12 compares the features extracted from the performance sound by the user U with the features of the performance sounds for each level, and determines the performance level corresponding to the performance sound for each level having similar features as the performance level of the user U. The generation unit 12 may change the sound output from the programmed music piece according to the performance level of the user U. For example, the generation unit 12 stores information that associates the difficulty of playing with each phrase in a programmed piece of music. If the user U has a low playing level, i.e., is close to a beginner level, the generation unit 12 mutes only phrases with low difficulty in the phrases in the programmed piece that use the timbre of an instrument corresponding to the instrument information. This allows easy phrases that a beginner-level user U can play to be played live, while difficult phrases can be played in a programmed piece of music.

[0054] Furthermore, the generation unit 12 may change the instrument used in the programmed music piece to a different instrument depending on the performance actions of the user U. For example, if the performance actions of the user U indicate that a string instrument is being played, the generation unit 12 changes a part played on piano in EDM to a part played on a string instrument. This makes it possible to generate a piece of music that blends different genres that are not normally used. By playing a part of a piece of music that combines unexpected instruments that are not normally used, the user U can have an unexpected and interesting experience and enjoy playing.

[0055] FIG. 4 is a flowchart showing the flow of processing performed by the information processing device 10 in this embodiment. First, the information processing device 10 determines whether a performance motion has been detected (step S20). If a performance motion has been detected, the information processing device 10 stores the detected performance motion in the performance motion information 140 (step S21). The information processing device 10 acquires music information indicating the music being programmed (step S22). The information processing device 10 generates performance information by changing the music information based on the performance motion (step S23). Here, the generation unit 12 may generate performance information by changing the music information based on the performance sound corresponding to the performance motion or the characteristics of the performance sound. The information processing device 10 stores the generated performance information in the performance information 142 (step S24). The information processing device 10 outputs the performance information (step S25).

[0056] As described above, in the information processing device 10 according to the second embodiment, the detection unit 11 detects a performance action, which is a user's performance action, as a creative action. The generation unit 12 generates, as the performance information, a sound that is output from a pre-created programmed piece of music, modified in accordance with the performance action. This allows the information processing device 10 according to the second embodiment to change the programmed piece of music in accordance with the user U's performance action, thereby creating a piece of music that combines the programmed piece of music with the performance by the user U. Therefore, programmed music such as EDM can be arranged with live performances to liven things up.

[0057] Third Embodiment Here, a third embodiment will be described. In this embodiment, the information processing device 10 functions as a teacher.

[0058] (Teacher's Aspect 1) For example, the information processing device 10 is a device that determines whether or not the user U is interested in a musical instrument. For example, in conventional music schools, the instrument to be learned is often decided from the beginning, and an expert on that instrument is often the teacher. In many cases, students attend a music school chosen by their parents without having a chance to carefully consider what instrument they are interested in. However, it would be better to value the opportunity for students to begin learning a musical instrument more. This is because whether or not a student is interested in an instrument greatly affects the level of their subsequent learning.

[0059] To address this issue, in this embodiment, the information processing device 10 determines whether the user U is interested in an instrument based on the user U's playing movements. The generation unit 12 generates a degree of interest using, for example, a trained model. The degree of interest is the degree to which the user U is interested in an instrument. In this case, the trained model is a model trained using a training dataset in which labels (OK or NG) are associated with playing movements of a specific instrument. The OK label is a label associated with playing movements determined to indicate interest in the instrument. The NG label is a label associated with playing movements determined to indicate no interest in the instrument. Whether or not the user is interested in an instrument is determined, for example, by an expert on that instrument. By training using such a training dataset, the trained model trains the correspondence between playing movements and whether or not the user is interested. By repeating the training, the trained model becomes able to accurately predict whether or not the user U is interested in an instrument based on untrained playing movements.

[0060] The generation unit 12 inputs the performance actions of the user U playing the musical instrument into the trained model. The trained model predicts whether or not the user U is interested in the musical instrument based on the input performance actions and outputs the prediction. The generation unit 12 uses the prediction result output by the trained model as performance information. As a result, the prediction result of whether or not the user U is interested in the musical instrument is output from the information processing device 10 via the output unit 13.

[0061] (Teacher's Aspect 2) For example, the information processing device 10 is an electronic musical instrument that outputs sample performance sounds to help the user U improve his / her performance skills. In this case, for example, the information processing device 10 is a musical instrument played by the user U. The information processing device 10 arranges the sample performance to a level that the user is likely to be able to play, and outputs the arranged performance, in accordance with the user's performance.

[0062] Students' playing often contains various habits, and trying to force them to sound closer to the ideal by strictly correcting them often does not work. Respecting their playing habits while working towards the ideal sound is more likely to improve their playing skills without putting excessive strain on them.

[0063] From this perspective, the information processing device 10 outputs, as performance information, performance sounds (model notes) that the user can play relatively easily from among performance sounds (model notes) that serve as models for improving the performance skills of the user U. This makes it possible to provide the user U with performance sounds that are easy for the user to play and that can improve the performance skills as performance information.

[0064] The model note is, for example, a performance note that is closer to an ideal sound (ideal sound) than the performance note of the user U and has characteristics similar to the performance motion of the user U. The model note is a performance note that is close to the ideal sound and is played with a motion similar to the performance motion of the user U. By setting the model note to a sound that is close to the performance note of the user U, a sound that is easy for the user U to play can be presented. Furthermore, by setting the model note to a sound that is close to the ideal sound, the user U can be presented with a sound that is close to the ideal sound, that is, a performance note that will improve their performance. Therefore, a performance note that will allow the user U to immediately improve their performance can be presented as the model note.

[0065] For example, the generation unit 12 generates the model sounds using a feature space, which is a space in which features are represented by vectors.

[0066] As the performance motion features, for example, explicit features such as the magnitude and speed distribution of the motion, which can be extracted by statistical processing or general arithmetic processing, can be used. Furthermore, as the performance motion features, latent features, for example, features expressed by multidimensional latent vectors extracted by a classifier (a classifier that clusters motions) generated using a machine learning technique, can also be used. It is also possible to use a combination of both explicit features extracted by statistical processing or general arithmetic processing and latent features extracted using a machine learning technique.

[0067] The performance sound features may be explicit features that can be extracted by statistical processing or general arithmetic processing, such as the frequency characteristics or smoothness of the performance sound. Alternatively, the performance sound features may be latent features, such as features expressed by multidimensional latent vectors extracted using a classifier (a classifier that clusters performance sounds) generated using a machine learning technique. A combination of both explicit features extracted by statistical processing or general arithmetic processing and latent features extracted using a machine learning technique may also be used.

[0068] For example, the generation unit 12 stores in advance sound information of an ideal tone for each piece of music. The ideal tone is a performance tone obtained by playing a piece of music faithfully to the musical score using, for example, a music sequencer. The generation unit 12 also stores in advance sound information of a plurality of sample tones for each piece of music. The sample tones are, for example, performance tones of an expert. For each sample note, the generation unit 12 stores, together with the sound information, the performance actions taken when the sample note is played.

[0069] The generation unit 12 specifies the position coordinates of the ideal sound in the feature space based on features extracted from the sound information of the ideal sound. The generation unit 12 also specifies the position coordinates of each sample sound in the feature space based on features extracted from the sound information of each sample sound. This forms a feature space (first feature space) in which the ideal sound and each sample sound are mapped.

[0070] FIG. 5 is a schematic diagram showing an example of a first feature space. The vertical and horizontal axes of FIG. 5 indicate some feature related to the sound associated with the performance. While the example of FIG. 5 shows a two-dimensional feature space for ease of understanding, an actual feature space may be a multidimensional feature space with more dimensions than two. As shown in FIG. 5 , an ideal sound, multiple sample sounds, and a performance sound of user U can be mapped into the first feature space. In this example, the sample sounds mapped inside a circle E1 with the ideal sound at its center and the connecting line L1 connecting the ideal sound and the performance sound as its radius are closer to the ideal sound than the performance sound of user U and are performance sounds performed by a player who is better than user U. On the other hand, the sample sounds mapped outside the circle E1 are further from the ideal sound than the performance sound of user U and are performance sounds performed less well than user U.

[0071] The generation unit 12 also specifies position coordinates of the performance actions of the user U in the feature space based on the features extracted from the performance actions of the user U. The generation unit 12 also specifies position coordinates of each of the performance actions of the sample notes in the feature space based on the features extracted from the performance actions of the sample notes. This forms a feature space (second feature space) in which the performance notes of the user U and each of the performance actions of the sample notes are mapped.

[0072] FIG. 6 is a schematic diagram showing an example of the second feature space. The vertical and horizontal axes in FIG. 6 indicate some features related to the performance motion. In the example of FIG. 6, a two-dimensional feature space is shown for ease of understanding, but the actual feature space may be a multidimensional feature space with more dimensions than two. As shown in FIG. 6, multiple sample notes and the performance motions of each of the notes played by user U can be mapped into the second feature space. For example, sample notes mapped inside a circle E2 centered on user U's performance motion and having a radius of a predetermined distance are performance notes played by a performance motion similar to user U's performance motion. On the other hand, sample notes mapped outside circle E2 are performance notes played by a performance motion dissimilar to user U's performance motion.

[0073] The generation unit 12 identifies position coordinates of the playing motion in the second feature space based on the features extracted from the playing motion of the user U. The generation unit 12 maps the playing motion into the second feature space based on the identified position coordinates. The generation unit 12 acquires playing motions that are mapped within a predetermined range from the position at which the playing motion of the user U is mapped in the second feature space as playing motions similar to the playing motion of the user U (similar motions).

[0074] The generation unit 12 identifies position coordinates of a performance action in a first feature space based on features extracted from a performance sound corresponding to the performance action of the user U. The generation unit 12 maps a performance sound corresponding to the performance action to the first feature space based on the identified position coordinates. The generation unit 12 calculates a distance (first distance) from a position where the performance sound of the user U is mapped to a position where the ideal sound is mapped in the first feature space. The generation unit 12 also calculates a distance (second distance) from a position where a sample note corresponding to the similar action is mapped to a position where the ideal sound is mapped in the first feature space. The generation unit 12 compares the first distance with the second distance, and if the second distance is smaller than the first distance, sets the sample note located at the second distance as the model note to be presented to the user U. In this way, the generation unit 12 can set a sample note that is close to the performance action of the user U and closer to the ideal sound than the performance of the user U as the model note.

[0075] 7 is a flowchart showing the flow of processing performed by the information processing device 10 in this embodiment. This flowchart shows an example of a performance action that focuses on fingering (fingering action) in performance.

[0076] First, the information processing device 10 determines whether a performance movement has been detected (step S30). If a performance movement has been detected, the information processing device 10 stores the detected performance movement in the performance movement information 140 (step S31). The information processing device 10 stores performance sounds corresponding to the performance movement in the performance sound information 141 (step S32). The information processing device 10 extracts fingering movements from the performance movement (step S33). The information processing device 10 extracts fingering movements, for example, by extracting fingering movements from the performance movement.

[0077] The information processing device 10 acquires a first feature space as a feature space of sounds (step S34). Ideal sounds and sample notes are each mapped into the first feature space based on the sound features. The information processing device 10 maps performance sounds corresponding to performance actions into the first feature space (step S35). The information processing device 10 acquires a second feature space as a feature space of actions (step S36). Fingering actions for each sample note are mapped into the second feature space based on the action features. The information processing device 10 maps the fingering actions of the user U into the second feature space (step S37).

[0078] The information processing device 10 acquires, in the second feature space, sample notes played with fingering movements similar to those of the user U (step S38). In the second feature space, the information processing device 10 acquires fingering movements that are within a predetermined range (first range) from the fingering movements of the user U as sample notes played with fingering movements similar to those of the user U.

[0079] The information processing device 10 determines whether or not the sample note acquired in step S38 is mapped closer to the ideal note in the first feature space than the performance note of the user U (step S39). The information processing device 10 compares the distance (first distance) from the performance note of the user U to the ideal note in the first feature space with the distance (second distance) from the sample note acquired in step S38 to the ideal note, thereby determining whether or not the sample note is mapped closer to the ideal note in the first feature space than the performance note of the user U.

[0080] If the sample note is mapped close to the ideal note, the information processing device 10 sets the sample note as the sample note (step S40). The information processing device 10 stores the sample note acquired in step S40 as performance information in the performance information 142 (step S41). The information processing device 10 outputs a performance note corresponding to the performance action (step S42). The information processing device 10 outputs the performance information that is the sample note (step S43). By outputting the performance note corresponding to the performance action and the performance information that is the sample note, the user U can compare the performance note that he or she played with the sample note.

[0081] Furthermore, in step S43, the information processing device 10 may output the model notes and display fingering movements corresponding to the model notes. In this case, the fingering movements corresponding to the model notes are an example of "performance information." The information processing device 10 includes a display unit (not shown). The display unit of the information processing device 10 displays, for example, a musical score in which finger numbers are assigned to notes of a phrase corresponding to the model notes. This allows the user U to understand the fingering used to play the model notes. Since the model notes are played with fingering similar to the user U's playing, the user U can easily incorporate the fingering of the model notes and efficiently improve their playing.

[0082] On the other hand, in step S39, if the sample note is not mapped close to the ideal note, the information processing device 10 returns to step S38 and re-acquires the sample note. When re-acquiring the sample note, the range of fingering movements similar to that of the user U may be changed to a larger range. In this case, the information processing device 10 sets the predetermined range (first range) in step S38 to a larger range, and widens the search range for fingering movements similar to that of the user U, so that it can acquire a sample note played with fingering movements similar to that of the user U.

[0083] (Teacher's Aspect 3) For example, the information processing device 10 is a device that suggests fingerings for playing. The information processing device 10 suggests fingerings that may suit the user U, for example, in accordance with the fingerings used as the playing movements of the user U.

[0084] Generally, various fingerings are used in musical performances depending on the performer's finger length, hand flexibility, and the like. In order to express themselves in a unique way, performers may change their fingerings in their performances. For wind instruments such as the flute and piccolo, fingerings may differ depending on the instrument, for example, in the highest register. When a performer tries to devise a performance using different fingerings, it is not realistic to consider multiple fingering patterns for each phrase. From this perspective, if the information processing device 10 can suggest fingerings that are suitable for the user U in accordance with the user U's fingering, it would be beneficial for the user who wants to devise fingerings for their performances.

[0085] The generation unit 12 stores, in advance, a database of fingering information that associates various fingerings with phrases in a piece of music. The fingering information can be generated, for example, based on images of fingering movements performed by various performers.

[0086] The generation unit 12 maps the fingering movements extracted from the performance movements of the user U into the second feature space. The information processing device 10 identifies a phrase corresponding to the performance movements of the user U and acquires fingering information corresponding to the identified phrase. The generation unit 12 maps the fingering movements corresponding to the acquired fingering information into the second feature space. The generation unit 12 generates, as a performance movement, fingering information that has the smallest distance from the performance movements of the user U among the fingering information mapped into the second feature space. The fingering information with the smallest distance can be said to be fingering information performed with movements that are most similar to the performance movements of the user U. In other words, it is considered to indicate fingering that is most likely to suit the user U. Therefore, it is possible to present the fingering variations that are most likely to suit the user U.

[0087] The generating unit 12 may generate, as performance information, information that associates fingering information with the distances therebetween. This makes it possible to present fingering variations together with the degree to which they suit the user U. Therefore, the user U can try out various fingerings after understanding how well each fingering variation suits him or her.

[0088] As described above, in the information processing device 10 according to the third embodiment, the detection unit 11 detects a performance action, which is a user's performance action, as a creative action. The generation unit 12 generates, as performance information, information that brings the performance sound closer to the ideal sound, depending on the degree of difference between the characteristics of the performance sound corresponding to the performance action and the ideal sound that the user wishes to master. For example, the generation unit 12 calculates a connection line L1 that connects the ideal sound and the performance sound of the user U in the first feature space. The generation unit 12 calculates the distance from each sample note to the connection line L1 in the first feature space. The generation unit 12 determines, as the model note, the sample note that is closer to the ideal sound than the performance sound of the user U, among the sample notes whose distance to the connection line L1 is within a specific range. For example, through this processing, the generation unit 12 generates, as performance information, information that brings the performance sound closer to the ideal sound, depending on the degree of difference between the characteristics of the ideal sound that the user wishes to master. As a result, the information processing device 10 according to the third embodiment can present information that brings the performance sound closer to the ideal sound, for example, a sample sound that is better than the performance of the user U.

[0089] Furthermore, in the information processing device 10 according to the third embodiment, the generation unit 12 detects the user's fingering as a performance action. The generation unit 12 generates, as performance information, a model note that the user can finger and that is close to the ideal note, according to the characteristics of the fingering. This allows the information processing device 10 according to the third embodiment to present information that makes it easy for the user U to play and that brings the performance note closer to the ideal note.

[0090] (Fourth Embodiment) Here, a fourth embodiment will be described. In this embodiment, the information processing device 10 is an anthropomorphized device. For example, the information processing device 10 grasps the psychological state of the user U in response to the performance action of the user U as if the user U had a personality, and outputs information in line with the feelings of the user U as performance information.

[0091] For example, the information processing device 10 stores the performance history of the user U. For example, the information processing device 10 stores, as the performance history, information associating the performance sounds and performance actions that the user U has listened to or performed from the time the user U was born until the present with the time when the performance actions were performed (for example, the date and time when the performance actions were performed). The performance actions here include the user U's actions of listening to music as well as the user U's actions of playing music.

[0092] Based on the playing motion, the generation unit 12 determines whether the user U wants someone to play with him / her as the psychological state of the user U. For example, if the playing motion of the user U is different from usual motion, the generation unit 12 determines that the user U wants someone to play with him / her. On the other hand, if the playing motion is the same as usual motion, the generation unit 12 determines that the user U does not want someone to play with him / her.

[0093] Based on the performance history, the generation unit 12 calculates the degree to which the performance actions of the user U differ from the past performance action patterns of the user U. For example, the performance action patterns may be the frequency with which the user U plays, the length of time they play, the time of day they play, etc. For example, if the user used to play at least once a day but has not been playing recently, this deviates from the past performance action patterns of the user U, and the degree of difference is high.

[0094] The generation unit 12 calculates a pattern of past playing actions (first pattern). For example, the generation unit 12 acquires a group of playing actions performed in the past (first group of playing actions) from the performance history. For example, a plurality of playing actions performed during a first specific period prior to the present (e.g., a period from five months ago to three months ago) is set as the first group of playing actions. Using the first group of playing actions, the generation unit 12 calculates a playing pattern during the first specific period, such as the frequency of playing, the length of time for playing, and the time period for playing, and sets this as the first pattern.

[0095] The generation unit 12 calculates a pattern of recent playing actions (second pattern). For example, the generation unit 12 acquires a group of playing actions that have been recently performed (second group of playing actions) from the playing history. For example, the generation unit 12 sets the second group of playing actions to a plurality of playing actions performed during a second specific period including the present (e.g., a period from one month ago to the present). Using the second group of playing actions, the generation unit 12 calculates a playing pattern during the second specific period, such as the frequency of playing, the length of time for playing, and the time period for playing, and sets the calculated pattern as the second pattern.

[0096] The generation unit 12 compares the first pattern with the second pattern and calculates the degree of difference (dissimilarity) between the two patterns. For example, the generation unit 12 calculates the dissimilarity for each of the following items: the frequency of performance, the length of performance time, and the time period during which the performance is performed. Any method may be used to calculate the dissimilarity. For example, the dissimilarity can be calculated using a difference table in which the difference between the patterns and the difference level are associated. The difference table associates the difference with the difference level, for example, if the frequency of performance per week is less than half a day, difference level 2 if the frequency is more than half a day but less than two days, and difference level 3 if the frequency is two days or more. The generation unit 12 calculates the dissimilarity level for each of the following items: the frequency of performance per week, the length of performance time, and the time period during which the performance is performed, using the difference table, and then calculates the dissimilarity by adding up the calculated dissimilarity levels.

[0097] When the degree of difference is equal to or greater than a threshold, the generation unit 12 determines that the user U wants someone to pay attention to him / her. On the other hand, when the degree of difference is less than the threshold, the generation unit 12 determines that the user U does not want someone to pay attention to him / her.

[0098] When the generating unit 12 determines that the user U wants someone to play with him / her, it generates performance information. The generated performance information is performance sounds that the user U has listened to or played in the past. By using past performance sounds as performance information, the user U can listen to the performance sounds that he / she has listened to or played in the past, and can reminisce about the past or feel nostalgic. This makes it possible to output information that is in line with the user U's feelings as performance information.

[0099] For example, the generating unit 12 generates, as performance information, a performance sound selected from performance sounds corresponding to performance actions performed in the past in accordance with a recently performed performance action.

[0100] The generation unit 12 acquires performance sounds (first performance sounds) corresponding to each group of performance actions (first performance action group) performed in the past from the performance history. For example, the first performance sounds are performance sounds corresponding to each of a plurality of performance actions performed during a first specific period prior to the present (e.g., a period from five months ago to three months ago). The generation unit 12 sets a nostalgia level for each of the first performance sounds. The nostalgia level represents the degree of nostalgia. For example, songs that the user U often listened to in the past or songs that the user practiced over and over again are likely to be memorable and cause a high degree of nostalgia for the songs. Based on this concept, the generation unit 12 classifies each of the first performance sounds according to their frequency of occurrence within the first specific period and sets a nostalgia level according to the classification result.

[0101] The generation unit 12 performs clustering by classifying the first performance sounds into a plurality of groups based on similar performance sounds. Any method may be used for clustering. For example, based on the similarity between a certain performance sound and another performance sound, similar performance sounds may be classified into a plurality of groups. The similarity between the performance sounds may be calculated based on the distance between the performance sounds in a feature space (first feature space) of the performance sounds, for example.

[0102] The generation unit 12 calculates the number of performance notes included in each clustered group and identifies the group that includes the most performance notes. The generation unit 12 sets the nostalgia level of the performance notes included in the identified group to the highest value, thereby identifying the performance notes that evoke the most nostalgia. In this way, the generation unit 12 sets the nostalgia level according to the number of performance notes included in the group.

[0103] The generation unit 12 selects a nostalgia level based on the degree of difference (degree of difference) between the first pattern and the second pattern. For example, the larger the degree of difference, the more nostalgic the selected nostalgia level. The generation unit 12 generates performance sounds of a group corresponding to the selected nostalgia level as performance information.

[0104] FIG. 8 is a flowchart showing the flow of processing performed by the information processing device 10 in this embodiment.

[0105] First, the information processing device 10 determines whether a performance action has been detected (step S50). If a performance action has been detected, the information processing device 10 stores the detected performance action in the performance action information 140 (step S51). The information processing device 10 stores performance sounds corresponding to the performance action in the performance sound information 141 (step S52). The information processing device 10 stores a performance history (step S53). Using the performance history, the information processing device 10 calculates a degree of difference between a past performance action pattern (first pattern) and a recent performance action pattern (second pattern) (step S54).

[0106] The information processing device 10 determines whether the degree of difference calculated in step S54 is equal to or greater than a threshold value (step S55). If the degree of difference calculated in step S54 is equal to or greater than the threshold value, the information processing device 10 determines to generate performance information. On the other hand, if the degree of difference calculated in step S54 is less than the threshold value, the information processing device 10 determines not to generate performance information.

[0107] If the degree of difference calculated in step S54 is equal to or greater than the threshold and it is determined that performance information should be generated, the information processing device 10 performs clustering to classify the past performance notes (first performance notes) into multiple groups using the performance history (step S56). The information processing device 10 selects one of the multiple groups classified in step S56 based on the degree of difference calculated in step S54 (step S57). The information processing device 10 sets a nostalgia level for each group depending on the number of performance notes included in each group. The information processing device 10 selects a group corresponding to the nostalgia level based on the degree of difference.

[0108] The information processing device 10 generates performance information from the performance sounds included in the selected group (step S58). The information processing device 10 stores the performance information generated in step S58 in the performance information 142 (step S59). The information processing device 10 outputs the performance information (step S60). In this embodiment, the degree of difference between past (e.g., from five months ago to three months ago) and recent (e.g., from one month ago to the present) performance actions is defined as the dissimilarity. In other words, even if no performance action is detected in step S50, the dissimilarity can be calculated as long as past and recent performance actions are accumulated. Therefore, in the flow of FIG. 8 , if step S50 is determined to be NO, the process shown in step S54 may be executed to calculate the dissimilarity based on past and recent performance actions.

[0109] As described above, in the information processing device 10 according to the fourth embodiment, the detection unit 11 detects a performance action, which is a user's performance action, as a creative action. The generation unit 12 stores, as a performance history, information associating performance sounds corresponding to the performance action with the performance action and the time when the performance action was performed. The generation unit 12 selects a performance sound from a first performance sound (an example of a performance sound corresponding to a performance action performed in a first specific period prior to the present) based on the degree of difference (an example of a performance action performed in a second specific period including the present). The generation unit 12 generates the selected performance sound as performance information. Thus, the information processing device 10 according to the fourth embodiment can, based on the performance history, detect whether the user U is acting differently than usual, for example, and whether the user U wants attention. Then, by outputting past performance sounds at a time when the user U wants attention, it is possible to have the user U reminisce about the past or reminisce about nostalgia.

[0110] (Modification of the Fourth Embodiment) A modification of the fourth embodiment will be described. In this modification, the information processing device 10 as a musical instrument is owned by different users U for different periods of time, just as a musical instrument is passed down from generation to generation.

[0111] In this modified example, the detection unit 11 of the information processing device 10 stores information as a performance history that associates the playing actions of the owner, user U, with the time when the playing actions were performed (e.g., the date and time when the playing actions were performed).

[0112] The generating unit 12 of the information processing device 10 extracts features of the performance sounds in the performance history, and maps the extracted features into a feature space (feature space of the performance sounds).

[0113] The performance sound features can be explicit features that can be extracted by statistical processing or general arithmetic processing, such as the frequency characteristics of the performance sound, the smoothness of the sound, the volume, the pitch, the speed (tempo) of the performance, and the timbre. Alternatively, the performance sound features can be latent features, such as features expressed by multidimensional latent vectors extracted using a classifier (a classifier that clusters performance sounds) generated using a machine learning technique. It is also possible to use a combination of both explicit features extracted by statistical processing or general arithmetic processing and latent features extracted using a machine learning technique.

[0114] The generation unit 12 divides the feature space into specific time intervals (e.g., one month) on which the performance was performed, and for each divided time interval, identifies an area (user performance area) onto which the features of the performance sound performed by the user U are mapped. If the representative positions (e.g., center of gravity positions) in the user performance areas for each time interval are separated by a threshold value or more, the generation unit 12 determines that the owner of the information processing device 10 has changed from the previous user U (past owner) to another user U (current owner).

[0115] When the generation unit 12 determines that the owner of the information processing device 10 has changed, it generates performance information that reflects the characteristics of the performance by the previous owner. For example, when the generation unit 12 determines that the owner of the information processing device 10 has changed, it generates a character that is an alter ego of the previous owner. This character has the playing habits and characteristics of the previous owner, and outputs performance sounds that sound as if the previous owner were playing, in accordance with the playing actions of the current owner.

[0116] The generation unit 12 determines, based on the performance history, whether there is a specific pattern (habit) in the circumstances under which the past owner performed the instrument. The generation unit 12 acquires, for example, situation information indicating the circumstances under which the past owner performed the instrument, such as information indicating the date, day of the week, time of day, and weather when the instrument was performed. The generation unit 12 classifies the acquired situation information into several groups by clustering. Any clustering method may be used here, and for example, a classifier generated using a machine learning method (e.g., a support vector machine) may be used.

[0117] If there is a group (majority group) among the classified groups for which the number of situation information items belonging to the group is equal to or greater than a threshold, the generation unit 12 determines that there is a specific pattern in the situations in which the past owner performed the instrument. The generation unit 12 determines that there is a pattern (habit) of the past owner performing the instrument in the situation (e.g., date, day of the week, time of day, weather, etc.) indicated by the situation information included in the majority group. If the current situation corresponds to the pattern (habit) of the past owner performing the instrument, the generation unit 12 generates performance information, which includes sounds played by the past owner. The output unit 13 outputs the performance information (sounds played by the past owner) generated by the generation unit 12.

[0118] For example, if the situation indicated by the situation information included in the majority group indicates a specific time period of the day, it indicates that the past owner had the habit of playing music at that specific time period of the day. In this case, the information processing device 10 outputs the sound played by the past owner at that specific time period of the day. This allows user U (the current owner) to know that the past owner had the habit of playing music at that specific time period and to listen to the sound played by the past owner.

[0119] The generation unit 12 may also generate, as performance information, performance sounds (similar performance sounds) that sound as if a past owner were playing the information processing device 10. For example, the information processing device 10 outputs accompaniment sounds (similar performance sounds) that sound as if the past owner were playing a performance by the current owner. This allows the user U (current owner) to experience playing an ensemble with the past owner.

[0120] For example, the generation unit 12 generates a similar performance sound using two models: a generation model and a performance model. The generation model is a model that generates variation data. The variation data is information that represents factors that vary the performance of a piece of music according to the characteristics of the performance by a specific person (here, the past owner). The performance model is a model that generates performance data. The performance data is information that represents a performance in which the variation data is reflected in the performance of the piece of music. By reflecting the variation data in the performance of the piece of music, a performance sound can be generated that sounds as if it were being played by a past owner.

[0121] The generative model and the performance model are, for example, statistical prediction models generated using machine learning techniques, and are models related to a VAE (Variational Auto Encoder) configured using a neural network or the like. A VAE is a model that creates information similar to training data (similar information) by training the features of the training data. A VAE consists of an encoder and a decoder, and the encoder converts the training data into latent variables, and the decoder generates similar information using the latent variables.

[0122] In this modified example, the encoder of the VAE corresponds to the generative model. The training data is information representing the performance notes played by past owners extracted from the performance history. The latent variables are features of the performance notes played by past owners extracted from the performance history. The decoder of the VAE corresponds to the performance model. The similarity information is information (performance data) representing the performance notes played as if they were played by the past owner.

[0123] For example, suppose a past owner repeatedly played and practiced the accompaniment of a certain piece of music. In this case, the generation unit 12 generates latent variables by inputting the accompaniment sounds of that piece of music as training data into the generative model. Then, the generation unit 12 inputs the musical score data and the latent variables into the performance model, thereby generating information representing the accompaniment sounds as if the past owner were playing. The generation unit 12 stores the generated information (information representing the accompaniment sounds as if the past owner were playing) as performance data in the storage unit 14 as performance information 142.

[0124] When the performance action of user U (current owner) detected by the detection unit 11 is an action of playing a certain song, the generation unit 12 refers to the memory unit 14 to acquire performance data corresponding to the accompaniment of that song, and outputs the acquired performance data to the output unit 13.

[0125] 9 is a flowchart showing the flow of processing performed by the information processing device 10 in this embodiment. The processing relating to steps S90 to S93 shown in Fig. 9 is the same as steps S50 to S53 shown in Fig. 8, and therefore a description thereof will be omitted.

[0126] The information processing device 10 extracts the characteristics of the performance sounds related to the performance history (step S94). The information processing device 10 identifies an area (user performance area) to which the characteristics of the performance sounds performed by the user U are mapped for each specific time interval (step S95). The information processing device 10 calculates the distance (distance in the feature space) between the user performance areas for each time interval (step S96).

[0127] The information processing device 10 determines whether the distance between the user performance areas (distance in the feature space) is equal to or greater than a threshold value (step S97). If the distance between the user performance areas (distance in the feature space) is equal to or greater than the threshold value, the information processing device 10 determines that the owner of the information processing device 10 has changed (changed to another owner) (step S98).

[0128] When the owner of the information processing device 10 changes, the information processing device 10 generates performance information that reflects the characteristics of the performances made by the previous owner (step S99). For example, the information processing device 10 generates information that represents the performance sounds made by the previous owner, associated with the performance patterns (habits) made by the previous owner. Alternatively, the information processing device 10 generates information that represents the performance sounds (similar performance sounds) that sound as if the previous owner had performed them.

[0129] The information processing device 10 stores the performance information generated in step S99 in the performance information 142 (step S100). The information processing device 10 outputs the performance information (step S101). For example, the information processing device 10 outputs the performance sounds played by the past owner in the same pattern (habit) as the performance pattern of the past owner. Alternatively, the information processing device 10 outputs accompaniment sounds (similar performance sounds) that sound as if the past owner were playing, in response to the performance by the current owner.

[0130] As described above, in the modified example of the fourth embodiment, the detection unit 11 detects a performance action, which is a user's performance action, as a creative action. The generation unit 12 stores, as a performance history, information associating performance sounds corresponding to the performance action with the performance action and the time when the performance action was performed. The generation unit 12 determines whether the owner of the information processing device 10 has changed based on the performance history. If the generation unit 12 determines that the owner of the information processing device 10 has changed, it generates performance information reflecting the characteristics of performances by past owners. As a result, the information processing device 10 according to the modified example of the fourth embodiment can present the characteristics of performances by past owners to the current owner based on the performance history in response to a change in the owner of the information processing device 10 (the user). Therefore, the current owner can know who previously owned the instrument (information processing device 10) and how it was played.

[0131] Fifth Embodiment Here, a fifth embodiment will be described. In this embodiment, the information processing device 10 is a device that reflects feedback from a user U in the process of creative activities such as creating music. The feedback here indicates a direction for arranging the music, such as "8-beat rock style," "more happiness," or "more transparency."

[0132] For example, the information processing device 10 is an electronic musical instrument in which a music sequencer is installed, and outputs sounds corresponding to sound information of a piece of music generated using the music sequencer.

[0133] The user U generates a piece of music using the information processing device 10. The user U generates the piece of music by inputting sound information into a music sequencer or playing a keyboard. The user U provides feedback on the piece of music he or she has generated. For example, the user U verbally indicates the direction of arrangement of the piece of music he or she has generated by saying something like, "I want it to be an 8-beat rock style." The information processing device 10 regenerates the piece of music in accordance with the instructions (feedback) from the user U and outputs the performance sounds of the regenerated piece of music as performance information.

[0134] The information processing device 10 stores various texts, for example, on the Internet, etc., particularly texts related to music composition and music arrangement, in an analysis database. Furthermore, when a piece of music is associated with a piece of text, the information processing device 10 also stores sound information of the piece of music in association with the text in the analysis database.

[0135] The detection unit 11 detects feedback for the music. For example, the detection unit 11 has a function of collecting sound (e.g., a microphone) and a function of converting the voice into text information (e.g., voice recognition). The detection unit 11 collects the voice of the user U using the microphone and converts the collected voice into text information using voice recognition or the like. The detection unit 11 outputs the text information to the generation unit 12.

[0136] The generation unit 12 predicts a countermeasure for the feedback using the analytical database. The generation unit 12 extracts sentences that include the wording included in the text information from the sentences stored in the analytical database.

[0137] For example, the generation unit 12 performs morphological analysis on the words and phrases included in the text information to extract morphemes, such as nouns and adjectives, included in the text information. For example, the generation unit 12 extracts morphemes such as "8 beat," "rock style," "happy," and "transparent." Based on the extracted morphemes, the generation unit 12 references an analysis database and extracts sentences containing the morpheme "8 beat" and other words included in the text information from sentences stored in the analysis database. For example, the generation unit 12 extracts sentences such as "8 beat means...," "The classic 8 beat has..., and can be said to be a standard beat," and "...therefore, it results in a rock-style chord progression." The generation unit 12 predicts a countermeasure for the text information using the analysis results of the extracted sentences. For example, if the extracted sentences include phrases such as "The drums determine the beat of the song" and "The hi-hat plays eighth notes," the generation unit 12 predicts, for example, "Have the drum machine play eighth notes repeatedly" as a countermeasure for "8 beat." Alternatively, when a piece of music is associated with a sentence, sound information, such as tone, rhythm, or phrase, that is frequently used in the music or that is used in many pieces of music, is extracted. Such frequently used sound information is considered to form the text information "8 beat." In this case, the generation unit 12 predicts such frequently used sound information as a countermeasure for the "8 beat," for example.

[0138] The generation unit 12 regenerates (arranges) the music based on the information indicating the predicted countermeasures.

[0139] For example, if the countermeasure predicted by the generation unit 12 is text information such as "have the drum machine repeatedly play eighth notes," the text information indicating the countermeasure is converted into sound information corresponding to the content of the text information. In this case, for example, the information processing device 10 creates an arrangement table in advance. The arrangement table is a table that associates text information with sound information corresponding to the text information. For example, in the arrangement table, the text information "have the drum machine repeatedly play eighth notes" is associated with sound information that creates an "eight-beat" feel, such as outputting a specific rhythm (e.g., eighth notes) with a specific tone (e.g., hi-hat) in the drum machine. The generation unit 12 references the arrangement table based on the text information acquired from the generation unit 12 and acquires sound information associated with the text information. The generation unit 12 regenerates the music by adding the acquired sound information to the music. The generation unit 12 uses the regenerated sound information of the music as performance information.

[0140] For example, if the countermeasure predicted by the generation unit 12 is sound information, the generation unit 12 regenerates the music by adding the sound information to the music. The generation unit 12 sets the regenerated sound information of the music as performance information.

[0141] 10 is a flowchart showing the flow of processing performed by the information processing device 10 in this embodiment. First, the information processing device 10 determines whether feedback has been detected (step S70). If feedback has been detected, the information processing device 10 predicts a countermeasure for the feedback based on the detected feedback (step S71). The information processing device 10 regenerates a piece of music based on the prediction (step S72). The information processing device 10 adds the regenerated piece of music to the performance information 142 as performance information (step S73). The information processing device 10 outputs the generated performance information (step S74).

[0142] As described above, in the information processing device 10 according to the fifth embodiment, the detection unit 11 detects feedback on the music piece generated during the creative operation. The generation unit 12 generates, as performance information, a sound that recreates the music piece based on the feedback. As a result, in the information processing device 10 according to the fifth embodiment, as long as the user U indicates a direction for arranging the music piece, the information processing device 10 can recreate the music piece in accordance with that direction.

[0143] (Modification of the embodiment) A modification of the embodiment will be described. This modification can be applied to any of the above-described embodiments (first to fifth embodiments). In this modification, the output unit 13 outputs performance information via a character. The character here is an anthropomorphized object, for example, an object having an appearance that resembles a person or an animal. For example, the character is a person, a doll, a small animal, or a robot having a friendly appearance.

[0144] For example, the information processing device 10 may be a robot that outputs performance information from a speaker provided on the robot. Alternatively, the information processing device 10 may have a display, and when performance information is output, a character may be displayed on the display and controlled to move in conjunction with the performance information. In this way, the performance information may be output via a character, allowing the user U to feel a sense of familiarity with the information processing device 10.

[0145] 11 is a diagram illustrating a modification of the embodiment. As shown in FIG. 11, in this modification, performance information is output via a character C.

[0146] In this modified example, the configuration may be such that the characteristics of character C can be set. For example, the generation unit 12 sets the characteristics of character C according to the level of the performance sound corresponding to the performance action (performance level). For example, if the performance level is low and at a beginner level, the characteristics of character C are set to a child of about 2-3 years old. If the performance level is slightly higher and at an intermediate level, the characteristics of character C are set to a minor of about 15 years old. If the performance level is higher and at an advanced level, the characteristics of character C are set to an adult of about 25 years old. This allows character C to evolve as the performance level improves, allowing the user U to enjoy raising character C.

[0147] In the above description, the character C is developed in accordance with the performance level, but the present invention is not limited to this. The character C may be developed in accordance with the number of times or frequency of performance by the user U, in addition to the performance level.

[0148] As described above, in the information processing device 10 according to the modified embodiment, the output unit 13 outputs performance information via an anthropomorphized character. This allows the user U to feel a sense of familiarity with the information processing device 10. Furthermore, in the information processing device 10 according to the modified embodiment, the generation unit 12 sets the characteristics of the character according to the level of the performance sound corresponding to the performance action, and links fluctuations in the level with the characteristics of the character C. This allows the user U to enjoy developing the character C and encourages them to make an effort to improve their performance level.

[0149] All or part of the information processing device 10 in the above-described embodiment may be implemented by a computer. In this case, a program for implementing this function may be recorded on a computer-readable recording medium, and the program may be loaded into a computer system and executed. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. Furthermore, "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording medium" may also include media that dynamically store programs for a short period of time, such as communication lines used when transmitting programs via networks such as the Internet or telephone lines, or media that store programs for a fixed period of time, such as volatile memory within a computer system that serves as a server or client. The program may also be designed to implement part of the aforementioned functions, or may be capable of implementing the aforementioned functions in combination with a program already stored in the computer system, or may be implemented using a programmable logic device such as an FPGA.

[0150] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, as well as within the scope of the invention described in the claims and their equivalents.

[0151] Furthermore, the configurations of the above-described embodiments may be combined. For example, when the information processing device 10 generates a musical piece based on a performance action as performance information in the first embodiment, the information processing device 10 may reorganize the musical piece based on feedback, which is an instruction from the user U, and generate the reorganized musical piece as performance information. For example, the detection unit 11 detects a first performance action of the user and a second performance action that occurred after the first performance action. The generation unit 12 generates information based on the first performance action, i.e., performance information that can be used to create a creative object, as first performance information. The generation unit 12 generates second performance information by modifying the first performance information based on the second performance action. The output unit 13 outputs the second performance information as performance information that can be used to create a creative object. For example, the generation unit 12 generates a musical piece based on the performance action as performance information. The detection unit 11 further detects feedback, which is an instruction from the user, regarding the musical piece (the musical piece generated by the generation unit 12 based on the first performance action). When the detection unit 11 detects feedback or a second performance action, the generation unit 12 may regenerate a piece of music based on the information (the second performance action or the feedback) detected by the detection unit 11. The performance information may be at least performance information that can be used to create a creative subject. For example, the performance information may include at least one of the timbre and volume of a sound signal corresponding to a performance action, start or stop of recording, start or stop of time measurement, change of a display on a display unit of the information processing device 10, and generation of a piece of music.

[0152] 10...information processing device, 11...detection unit, 12...generation unit, 13...output unit

Claims

1. An information processing device comprising: a detection unit that detects a first playing action, which is a playing action performed by a user, as an action for creating an object of creation, and a second playing action, which is an action performed by the user that occurs after the first playing action; a generation unit that generates first playing information based on the first playing action, and second playing information that modifies the first playing information based on the second playing action; and an output unit that outputs the second playing information generated by the generation unit as performance information capable of creating the object of creation based on the playing actions.

2. The information processing device of claim 1, wherein the generation unit generates a musical piece using the first playing action, the detection unit detects feedback which is an instruction from the user regarding the musical piece generated by the generation unit, the generation unit regenerates a musical piece using the detected feedback or second playing action when the detection unit detects the feedback or the second playing action, and the output unit outputs the musical piece regenerated by the generation unit as the performance information.

3. An information processing device comprising: a detection unit that detects a performance action performed by a user as an action for creating an object of creation; a generation unit that generates, as performance information for creating the object of creation based on the performance action, information that brings the performance sound corresponding to the performance action closer to the ideal sound in accordance with the difference between the characteristics of the performance sound corresponding to the performance action and the ideal sound of the piece of music corresponding to the performance action; and an output unit that outputs the performance information.

4. An information processing device comprising: a detection unit that detects a performance action performed by a user as an action for creating an object of creation; a generation unit that generates, as performance information for creating the object of creation based on the performance action, a performance sound corresponding to the performance action performed in a second specific period including the present, selected from the performance history performed in a first specific period prior to the present, using a performance history that associates performance sounds corresponding to the performance action with the performance action and the time when the performance action was performed; and an output unit that outputs the performance information.

5. An information processing device as described in claim 1 or claim 2, wherein the performance information includes at least one of the tone and volume of the sound signal corresponding to the performance action, start or stop of recording, start or stop of time measurement, change of the display on a display unit of the information processing device, and generation of a musical piece.

6. The information processing device according to claim 1 or 2, wherein the generation unit generates, as the performance information, a sound that changes according to the body movement extracted from the performance movement.

7. An information processing device according to claim 1 or 2, wherein the generation unit generates, as the performance information, a sound that is output from a pre-created programmed piece of music and that is modified in accordance with the performance action.

8. The information processing device of claim 3, wherein the detection unit detects fingering movements made by the user when playing as the performance movements, the generation unit generates model notes that the user can finger and that are close to the ideal tone according to the characteristics of the fingering movements as the performance information, and the output unit outputs the model notes and displays the fingering corresponding to the model notes.

9. The information processing device according to any one of claims 1 to 4, wherein the output unit outputs the performance information via a personified character.

10. The information processing device according to claim 9, wherein the generation unit sets characteristics of the character according to a level of a performance sound corresponding to the performance action, and links fluctuations in the level with the characteristics.

11. An information processing device as described in claim 4, further comprising a memory unit that stores the performance history, wherein the generation unit generates the performance information using the performance history stored in the memory unit.

12. The information processing device of claim 4, wherein the generation unit uses the performance history to calculate a dissimilarity, which is the degree to which the user's current playing actions differ from the user's past playing actions, and if the calculated dissimilarity is equal to or greater than a threshold value, generates the user's past playing actions as the performance information.

13. The information processing device of claim 4, wherein the generation unit determines whether the owner of the information processing device has changed based on the performance history, and if it determines that the owner of the information processing device has changed, generates the performance information that reflects the characteristics of the performance by the past owner.

Images