Information processing device, information processing method, and program

Abstract

This information processing device comprises: an acquisition unit that acquires performance information, which indicates performance of a user, and / or behavior-related information, which is information different from the performance information and indicates information related to the behavior of the user; an information processing unit that generates output information, which is non-performance information responding to the request of the user and indicates information relating to music, on the basis of the acquired performance information and / or behavior-related information; and an output processing unit that outputs the output information.

Description

Information processing device, information processing method, and program

[0001] The present invention relates to an information processing device, an information processing method, and a program.

[0002] There is a need to find new value in musical instruments, for example, a sense of symbiosis, where the instrument is close to the player.

[0003] For example, Patent Document 1 listed below discloses a technology that assists a performer in their practice by inferring weak points in their playing style from the performance content of the performer and presenting music suitable for practicing that playing style.

[0004] JP 2009-157063 A

[0005] However, it is hoped that they will be able to coexist with music, not just by providing support in practicing performance, but also by providing support in other aspects besides performance.

[0006] The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an information processing device, an information processing method, and a program that can improve the sense of coexistence with music.

[0007] In order to solve the above-mentioned problems, one aspect of the present invention is an information processing device that includes an acquisition unit that acquires at least one of performance information that indicates a performance by a user and behavior-induced information that is information different from the performance information and indicates information that is caused by the user's behavior, an information processing unit that generates output information that is non-performance information in response to a request from the user and indicates information about music based on the acquired information from the performance information and the behavior-induced information, and an output processing unit that outputs the output information.

[0008] Another aspect of the present invention is an information processing method implemented by a computer that acquires at least one of performance information indicating a performance by a user and behavior-induced information that is information different from the performance information and indicates information attributable to the user's behavior, generates output information that is non-performance information in response to a request from the user and indicates information related to music, based on the acquired information from the performance information and the behavior-induced information, and outputs the output information.

[0009] Another aspect of the present invention is a program that causes a computer to acquire at least one of performance information indicating a performance by a user and behavior-induced information that is information different from the performance information and indicates information attributable to the user's behavior, generate output information that is non-performance information in response to a request from the user and indicates information related to music based on the acquired information from the performance information and the behavior-induced information, and output the output information.

[0010] According to the present invention, it is possible to improve the sense of coexistence with music.

[0011] FIG. 1 is a block diagram showing an example of the functional configuration of a smart musical instrument according to each embodiment. FIG. 2 is a block diagram showing an example of the functional configuration of a music generation server according to the first embodiment. FIG. 3 is a block diagram showing an example of the functional configuration of a reason estimation server according to the first embodiment. FIG. 4 is a sequence diagram showing an example of the flow of a music generation process according to the first embodiment. FIG. 5 is a sequence diagram showing an example of the flow of a generation reason estimation process according to the first embodiment. FIG. 6 is a block diagram showing an example of the functional configuration of a music generation server according to the second embodiment. FIG. 7 is a sequence diagram showing an example of the flow of a music generation process according to the second embodiment. FIG. 8 is a block diagram showing an example of the functional configuration of a communication server according to the third embodiment. FIG. 9 is a sequence diagram showing an example of the flow of a communication process according to the third embodiment. FIG. 10 is a block diagram showing an example of the functional configuration of an avatar management server according to the fourth embodiment. FIG. 11 is a sequence diagram showing an example of the flow of an avatar management process according to the fourth embodiment. FIG. 12 is a sequence diagram showing an example of the flow of past information output process according to the fifth embodiment. FIG. 13 is a sequence diagram showing an example of the flow of practice support process according to the sixth embodiment.

[0012] Embodiments of the present invention will be described in detail below with reference to the drawings. Smart musical instruments according to each embodiment will be described with reference to FIG. 1 . FIG. 1 is a block diagram showing an example of the functional configuration of a smart musical instrument according to each embodiment. The smart musical instrument 10 shown in FIG. 1 is an instrument having an information processing function and is an example of an information processing device. The information processing function of the smart musical instrument 10 is realized using, for example, AI (Artificial Intelligence). Note that the information processing device is not limited to the smart musical instrument 10 and may be, for example, a smartphone, a tablet device, a wearable device, or a PC (Personal Computer). As shown in FIG. 1 , the smart musical instrument 10 includes a communication unit 110, an input unit 120, a storage unit 130, a control unit 140, and an output unit 150.

[0013] (1) Communication Unit 110 The communication unit 110 has a function of transmitting and receiving various information. For example, the communication unit 110 is communicably connected to a sensor device via a wired connection or a wireless connection (e.g., short-range wireless communication) and transmits and receives various information between the smart musical instrument 10 and the sensor device. The communication unit 110 is also communicably connected to a server via a network and transmits and receives various information between the smart musical instrument 10 and the server.

[0014] (2) Input Unit 120 The input unit 120 has a function of receiving input from a user. The input unit 120 is configured, for example, by an input device (e.g., a button, a touch panel, a microphone, etc.) provided as hardware in the smart musical instrument 10, or the instrument part of the smart musical instrument 10.

[0015] (3) Storage Unit 130 The storage unit 130 has a function of storing various types of information. The storage unit 130 is configured by a storage medium provided as hardware in the smart musical instrument 10, such as a hard disk drive (HDD), a solid state drive (SSD), a flash memory, an electrically erasable programmable read-only memory (EEPROM), a random access read / write memory (RAM), a read-only memory (ROM), or any combination of these storage media.

[0016] (4) Control Unit 140 The control unit 140 has a function of controlling the overall operation of the smart musical instrument 10. The control unit 140 is realized, for example, by causing a central processing unit (CPU) or a graphics processing unit (GPU), which are included as hardware in the smart musical instrument 10, to execute a program. As shown in FIG. 1 , the control unit 140 includes an acquisition unit 141, an information processing unit 142, and an output processing unit 143.

[0017] (4-1) Acquisition Unit 141 The acquisition unit 141 has a function of acquiring various types of information. For example, the acquisition unit 141 acquires at least one of performance information and behavior-induced information. The performance information indicates a performance by a user. The behavior-induced information is information different from the performance information and indicates information caused by the user's behavior.

[0018] (4-2) Information Processing Unit 142 The information processing unit 142 has a function of performing information processing. For example, the information processing unit 142 generates output information based on the performance information and the behavior-induced information acquired by the acquisition unit 141. The output information is non-performance information in response to a user request, and indicates information related to music. The user request is, for example, a demand in response to the user's psychology.

[0019] (4-3) Output Processing Unit 143 The output processing unit 143 has a function of outputting output information. For example, the output processing unit 143 causes the output unit 150 to output the output information generated by the information processing unit 142.

[0020] (5) Output Unit 150 The output unit 150 has a function of outputting various information. The output unit 150 is configured, for example, by an output device provided as hardware in the smart musical instrument 10, such as a display device or a touch screen (touch panel), or an audio output device such as a speaker. The output unit 150 displays output information on the display device and outputs output information from the audio output device, for example, in accordance with control by the output processing unit 143.

[0021] The functional configuration of the smart musical instrument 10 has been described above. As described above, the smart musical instrument 10 includes an acquisition unit 141 that acquires at least one of performance information indicating a performance by the user and behavior-driven information, which is information different from the performance information and indicates information attributable to the user's actions. The information processing unit 142 generates output information indicating non-performance information related to music based on the acquired performance information and behavior-driven information, which is non-performance information corresponding to the user's psychology (request). The output processing unit 143 outputs the output information. With this configuration, the user can obtain non-performance information related to music from the smart musical instrument 10 in accordance with their psychology, even when not playing an instrument in daily life, and can feel that the smart musical instrument 10 is supporting them in their daily lives. Therefore, the smart musical instrument 10 can enhance the sense of coexistence with music. An embodiment of the smart musical instrument 10 will be described in detail below.

[0022] <<1. First Embodiment>> A first embodiment will be described with reference to FIGS. 2 to 5 . In the first embodiment, a music generation system using a smart musical instrument 10 will be described. The music generation system in the first embodiment is a system using AI, and is a system that uses AI to randomly generate music and estimate the reason for generating the music. Music generated by AI is generated, for example, from the voltage of electricity flowing through a user's body. Music generation by AI is performed using a music generation model. The music generation model is a trained model trained (machine learning) to generate music based on the relationship between the voltage of electricity flowing through a user's body and the music to be generated. Therefore, when voltage information indicating the voltage is input, the music generation model generates music according to the voltage information and outputs the generated music. The reason for generating the music estimated by AI is, for example, the user's emotions at the time of generating the music. The reason for generating the music by AI is estimated using a reason estimation model. The reason estimation model is a trained model trained (machine learning) to generate a generation reason based on the relationship between the music and the reason for generating the music. Therefore, when a piece of music is input, the reason estimation model estimates the user's emotions at the time of generating the piece of music and outputs the estimation result as a generation reason. Hereinafter, a first embodiment will be described using an example in which the smart musical instrument 10 outputs, as output information, the generation reason for a piece of music generated by AI in response to an input piece of music.

[0023] <1-1. Configuration of Musical Piece Creation System> The configuration of the musical piece creation system according to the first embodiment will be described. The musical piece creation system includes a smart musical instrument 10, a sensor device, a musical piece creation server, and a reason estimation server. The network used for transmitting and receiving information may be, for example, a local area network (LAN), a wide area network (WAN), a telephone network (such as a mobile phone network or a landline telephone network), a regional Internet Protocol (IP) network, or the Internet.

[0024] (1) Smart Musical Instrument 10 As described with reference to Fig. 1, the smart musical instrument 10 is a musical instrument having an information processing function and is an example of an information processing device. The smart musical instrument 10 is communicably connected to a sensor device via a wired or wireless connection. The smart musical instrument 10 is also communicably connected to a music generation server and a reason estimation server via a network.

[0025] A screen for using the music generation system is displayed on the smart musical instrument 10 by an application (hereinafter also referred to as a "music generation app") that allows the user to use the music generation system. The user can generate music, play music, check the reason for its generation, and so on, by operating the screen displayed on the smart musical instrument 10 by the music generation app. The functions of the music generation app may be provided by installing the music generation app on the smart musical instrument 10 (i.e., a native app), or may be provided by a web system (i.e., a web app). In the case of a web app, the music generation app is managed by a server, and its functions are provided via a web browser.

[0026] (2) Sensor Device The sensor device has a function of acquiring biometric information of the user. The sensor device is, for example, a voltage sensor that measures the voltage of electricity flowing through the user's body and acquires voltage information indicating the voltage as biometric information. The sensor device is communicably connected to the smart musical instrument 10 via a wired or wireless connection. The sensor device transmits the acquired biometric information to the smart musical instrument 10.

[0027] (3) Music Generation Server The music generation server is a server that generates music using AI. The music generation server is composed of one or more servers (e.g., cloud servers). The music generation server generates music according to the voltage of the electricity flowing through the user's body, for example, based on the user's voltage information. The music generation server is connected to the smart musical instrument 10 via a network so as to be able to communicate with the smart musical instrument 10.

[0028] (4) Reason Estimation Server The reason estimation server is a server that estimates the reason for the creation of a piece of music. The reason estimation server is configured with one or more servers (e.g., cloud servers). The reason estimation server estimates the reason for the creation of a piece of music, for example, a piece of music selected by a user from among pieces of music created by the music creation server. The reason estimation server is communicably connected to the smart musical instrument 10 via a network.

[0029] <1-2. Functional Configuration of Smart Musical Instrument> The configuration of the music generation system according to the first embodiment has been described above. Next, the functional configuration of the smart musical instrument 10 according to the first embodiment will be described. As described with reference to FIG. 1 , the smart musical instrument 10 includes a communication unit 110, an input unit 120, a storage unit 130, a control unit 140, and an output unit 150.

[0030] (1) Communication Unit 110 The function of the communication unit 110 according to the first embodiment will be described. In communication with the sensor device, the communication unit 110 receives biological information such as voltage information and hormone information. In communication with the music generation server, the communication unit 110 transmits a music generation request and voltage information and receives music information. The music information includes music generated by the music generation server and information about the music. In communication with the reason estimation server, the communication unit 110 transmits a request for estimation of a generation reason and a music to be estimated, and receives reason information indicating the reason for generation of the music to be estimated.

[0031] (2) Input Unit 120 The functions of the input unit 120 according to the first embodiment will be described. The input unit 120 accepts inputs such as a music generation operation, a music playback operation, and a target selection operation. A music generation operation is an operation that requests the generation of a music piece. A music playback operation is an operation that requests the playback of a music piece. A target selection operation is an operation that selects a music piece whose generation reason is to be estimated and requests the estimation of the generation reason.

[0032] (3) Storage Unit 130 The function of the storage unit 130 according to the first embodiment will be described below. The storage unit 130 stores voltage information, music information, reason information, and the like.

[0033] (4) Control Unit 140 The following describes the functions of the control unit 140 according to the first embodiment. As described with reference to FIG. 1 , the control unit 140 includes an acquisition unit 141, an information processing unit 142, and an output processing unit 143.

[0034] (4-1) Acquisition Unit 141 The function of the acquisition unit 141 according to the first embodiment will be described. The acquisition unit 141 acquires, as behavior-driven information, a song selected by a user's target selection operation from among songs generated (automatically generated) by the music generation server (AI). The behavior-driven information is information resulting from the user's behavior of selecting a song.

[0035] (4-2) Information Processing Unit 142 The functions of the information processing unit 142 according to the first embodiment will now be described. The information processing unit 142 performs processing for generating music and estimating the reason for generation. When a user performs a music generation operation, the information processing unit 142 generates music information as output information based on the relationship between the electrical voltage flowing through the user's body and the music to be generated. For example, the information processing unit 142 generates music information corresponding to the voltage information acquired by the acquisition unit 141 using a music generation model (trained model) that has been trained (by machine learning) to generate music information based on the relationship between voltage and the music to be generated. Specifically, the information processing unit 142 transmits a music generation request and voltage information from the communication unit 110 to the music generation server, requesting music generation. As a result, the music generation server generates music using the music generation model. After the request, the information processing unit 142 stores the music information received by the communication unit 110 from the music generation server in the storage unit 130. The information processing unit 142 generates output information from the music information.

[0036] When a user performs a target selection operation, the information processing unit 142 generates information indicating the reason for the generation of the music information by adding it to the music information based on the reason for the generation of the music information. The information processing unit 142 generates reason information as output information based on the relationship between the music generated by the music generation server (AI) and the reason for the generation of the music. For example, the information processing unit 142 generates reason information for the music, which is behavior-driven information acquired by the acquisition unit 141, using a reason estimation model (trained model) trained (by machine learning) to generate reason information from the relationship between the music and the reason for the generation of the music. Specifically, the information processing unit 142 transmits a request for estimation of the reason for generation and the music, which is behavior-driven information, from the communication unit 110 to the reason estimation server, requesting estimation of the reason for the generation of the music. As a result, the reason estimation server estimates the reason for the generation of the music using the reason estimation model. After the request, the information processing unit 142 stores the reason information received by the communication unit 110 from the reason estimation server in the storage unit 130. The information processing unit 142 generates output information from the reason information. In this way, when the information processing unit 142 receives a song selected by the user from among the songs generated by AI (information resulting from the user's action of selection), it outputs the reason why the song was generated (information corresponding to the user's desire to know the reason for the generation).

[0037] (4-3) Output Processing Unit 143 The function of the output processing unit 143 according to the first embodiment will be described. The output processing unit 143 performs processing for displaying music information, playing music, displaying reason information, and the like. When a user performs a music generation operation, the output processing unit 143 causes the output unit 150 to display information (music information) about the music generated by the music generation server. When a user performs a music playback operation, the output processing unit 143 causes the output unit 150 to output the playback sound of the music selected by the user. When a user performs a target selection operation, the output processing unit 143 causes the output unit 150 to display the reason for generating the music estimated by the reason estimation server (reason information).

[0038] (5) Output Unit 150 The function of the output unit 150 according to the first embodiment will be described. The output unit 150 displays information about the music (music information) and reason information on a display device. The output unit 150 also outputs the playback sound of the music from an audio output device.

[0039] <1-3. Functional Configuration of the Music Generation Server> The functional configuration of the smart musical instrument 10 according to the first embodiment has been described above. Next, the functional configuration of the music generation server according to the first embodiment will be described with reference to FIG. 2. FIG. 2 is a block diagram showing an example of the functional configuration of the music generation server according to the first embodiment. As shown in FIG. 2, the music generation server 1030 includes a communication unit 1310, a storage unit 1320, and a control unit 1330.

[0040] (1) Communication Unit 1310 The communication unit 1310 has a function of transmitting and receiving various information. For example, the communication unit 1310 receives a music generation request and voltage information from the smart musical instrument 10 via the network, and transmits music information.

[0041] (2) Storage Unit 1320 The storage unit 1320 has the function of storing various types of information. The storage unit 1320 is configured by a storage medium provided as hardware in the music generation server 1030, such as an HDD, SSD, flash memory, EEPROM, RAM, ROM, or any combination of these storage media. As shown in FIG. 2 , the storage unit 1320 stores a music generation model 1321. The music generation model 1321 is a model (trained model) trained (by machine learning) to generate music based on the relationship between the voltage indicated by voltage information and the music to be generated. Therefore, the music generation model 1321 can receive voltage information as input and generate and output music corresponding to the voltage indicated by the voltage information.

[0042] (3) Control Unit 1330 The control unit 1330 has a function of controlling the overall operation of the music generation server 1030. The control unit 1330 is realized, for example, by causing a CPU or GPU provided as hardware in the music generation server 1030 to execute a program. As shown in FIG. 2 , the control unit 1330 includes a generation training unit 1331 and a music generation unit 1332.

[0043] (3-1) Generation Training Unit 1331 The generation training unit 1331 has a function of training in the generation of music. For example, the generation training unit 1331 trains to generate music based on the relationship between the voltage indicated by the voltage information and the music to be generated, and generates a music generation model 1321 that generates and outputs music corresponding to the voltage indicated by the input voltage information. The generation training unit 1331 stores the generated music generation model 1321 in the storage unit 1320.

[0044] (3-2) Music Generation Unit 1332 The music generation unit 1332 has a function of generating music information. For example, the music generation unit 1332 generates music information using the music generation model 1321. When the communication unit 1310 receives a music generation request and voltage information from the smart musical instrument 10, the music generation unit 1332 inputs the voltage information to the music generation model 1321 stored in the storage unit 1320. As a result, the music generation model 1321 generates and outputs music corresponding to the voltage. The music generation unit 1332 acquires the music generated by the music generation model 1321 and generates music information including information about the musical instrument. The music generation unit 1332 transmits the generated music information from the communication unit 1310 to the smart musical instrument 10.

[0045] <1-4. Functional Configuration of Reason Estimation Server> The functional configuration of the music generation server 1030 according to the first embodiment has been described above. Next, the functional configuration of the reason estimation server according to the first embodiment will be described with reference to FIG. 3. FIG. 3 is a block diagram showing an example of the functional configuration of the reason estimation server according to the first embodiment. As shown in FIG. 3, the reason estimation server 1040 includes a communication unit 1410, a storage unit 1420, and a control unit 1430.

[0046] (1) Communication Unit 1410 The communication unit 1410 has a function of transmitting and receiving various information. For example, the communication unit 1410 receives music information from the music generation server 1030 via the network. The communication unit 1410 also receives a request for estimating a generation reason and a music piece, which is behavior-causing information, from the smart musical instrument 10 via the network, and transmits reason information.

[0047] (2) Storage Unit 1420 The storage unit 1420 has a function of storing various information. The storage unit 1420 is configured by a storage medium provided as hardware in the reason estimation server 1040, such as an HDD, SSD, flash memory, EEPROM, RAM, ROM, or any combination of these storage media. As shown in FIG. 3 , the storage unit 1420 stores a reason estimation model 1421. The reason estimation model 1421 is a model (trained model) trained (by machine learning) to estimate a creation reason from the relationship between a song and the creation reason of the song. Therefore, the reason estimation model 1421 can input a song and estimate and output the creation reason of the song.

[0048] (3) Control Unit 1430 The control unit 1430 has a function of controlling the overall operation of the reason estimation server 1040. The control unit 1430 is realized, for example, by causing a CPU or GPU provided as hardware in the reason estimation server 1040 to execute a program. As shown in FIG. 3 , the control unit 1430 includes a reason training unit 1431 and a reason estimation unit 1432.

[0049] (3-1) Reason Training Unit 1431 The reason training unit 1431 has a function of training the estimation of the reason for creating a song. For example, the reason training unit 1431 trains to estimate a reason for creating a song from the relationship between the song and the reason for creating the song, and generates a reason estimation model 1421 that estimates and outputs the user's emotion corresponding to an input song as the reason for creating the song. As an example, the reason training unit 1431 uses a set of a song created by AI and the emotion of the user who requested the creation of the song at the time of song creation as a data set, and trains to generate a reason for creating the song from the relationship between the song and the reason for creating the song. The reason training unit 1431 stores the generated reason estimation model 1421 in the storage unit 1420.

[0050] (3-2) Reason Estimation Unit 1432 The reason estimation unit 1432 has a function of estimating the reason for the creation of a piece of music. For example, the reason estimation unit 1432 estimates the reason for the creation of a piece of music by using the reason estimation model 1421. When the communication unit 1410 receives a piece of music, the reason estimation unit 1432 inputs the piece of music to the reason estimation model 1421 stored in the storage unit 1420. As a result, the reason estimation unit 1432 estimates the reason for the creation of the piece of music by the reason estimation model 1421. The reason estimation unit 1432 generates reason information indicating the reason for the creation of the piece of music estimated by the reason estimation model 1421. The reason estimation unit 1432 transmits the generated reason information from the communication unit 1410 to the smart musical instrument 10.

[0051] 1-5. Processing Flow The functional configuration of the reason estimating server 1040 according to the first embodiment has been described above. Next, the processing flow according to the first embodiment will be described with reference to FIGS.

[0052] (1) Flow of Music Generation Process The flow of music generation process will be described with reference to Fig. 4. Fig. 4 is a sequence diagram showing an example of the flow of music generation process according to the first embodiment. As shown in Fig. 4, first, the sensor device 1020 measures the voltage of electricity flowing through the user's body (step S1101). After measurement, the sensor device 1020 transmits voltage information indicating the measured voltage value to the smart musical instrument 10 (step S1102).

[0053] The user performs a music generation operation to generate music on the smart musical instrument 10 (step S1103). When the input unit 120 of the smart musical instrument 10 receives the music generation operation input, the information processing unit 142 of the smart musical instrument 10 transmits voltage information and a generation request from the communication unit 110 to the music generation server 1030 (step S1104).

[0054] When the communication unit 1310 receives the voltage information and the generation request transmitted from the smart musical instrument 10, the music generation unit 1332 of the music generation server 1030 generates music (step S1105). Specifically, the music generation unit 1332 inputs the voltage information into the music generation model 1321 stored in the storage unit 1320 and acquires the music to be output. After the generation, the music generation unit 1332 transmits music information including the generated music from the communication unit 1310 to the smart musical instrument 10 (step S1106).

[0055] The output processing unit 143 of the smart musical instrument 10 causes the output unit 150 to display the music information received by the communication unit 110 from the music generation server 1030 (step S1107). The user performs a music playback operation on the smart musical instrument 10 to play a music piece (step S1108). When the input unit 120 accepts the music playback operation input, the information processing unit 142 of the smart musical instrument 10 plays the music piece selected by the user (step S1109).

[0056] (2) Flow of the Creation Reason Estimation Process The flow of the creation reason estimation process will be described with reference to Fig. 5. Fig. 5 is a sequence diagram showing an example of the flow of the creation reason estimation process according to the first embodiment. As shown in Fig. 5, first, the user performs a target selection operation on the smart musical instrument 10 to estimate the reason for creation of a song (step S1201). When the input unit 120 of the information processing unit 142 of the smart musical instrument 10 receives the input of the target selection operation, the information processing unit 142 transmits the song selected as the estimation target and an estimation request from the communication unit 110 to the reason estimation server 1040 (step S1202).

[0057] When the communication unit 1410 receives the song and the estimation request transmitted from the smart musical instrument 10, the reason estimation unit 1432 of the reason estimation server 1040 estimates the reason for generating the song (step S1203). Specifically, the reason estimation unit 1432 inputs the song to the reason estimation model 1421 stored in the storage unit 1420 and obtains reason information to be output. After the estimation, the reason estimation unit 1432 transmits the reason information from the communication unit 1410 to the smart musical instrument 10 (step S1204).

[0058] The output processing unit 143 of the smart musical instrument 10 causes the output unit 150 to display the reason information received by the communication unit 110 from the reason estimating server 1040 (step S1205). The user checks the generation reason displayed on the smart musical instrument 10 (step S1206).

[0059] As described above, the smart musical instrument 10 according to the first embodiment includes an acquisition unit 141 that acquires at least one of performance information indicating a performance by the user and behavior-induced information that is information different from the performance information and indicates information attributable to the user's behavior; an information processing unit 142 that generates output information indicating non-performance information that corresponds to the user's psychology (request) and information related to music based on the acquired information of the performance information and the behavior-induced information; and an output processing unit 143 that outputs the output information. The acquisition unit 141 acquires a song selected by the user from among songs generated by AI as behavior-induced information, and the information processing unit 142 generates information indicating the reason for generating the song, which is the acquired behavior-induced information, based on the relationship between the song generated by AI and the reason for generating the song, as output information.

[0060] With this configuration, even when the user is not playing a musical instrument in daily life, the user can input information resulting from the action of selecting a target song to the smart musical instrument 10, and obtain information from the smart musical instrument 10 that reflects the user's desire to know the reason for creating the song. This allows the user to feel that the smart musical instrument 10 is supporting the user as they create the song. Therefore, the smart musical instrument 10 according to the first embodiment can enhance the sense of coexistence with music.

[0061] <<2. Second Embodiment>> A second embodiment will be described with reference to FIGS. 6 and 7 . In the second embodiment, a music generation system using a smart musical instrument 10 will be described. The music generation system in the second embodiment is an AI-based system that generates music using AI. Music generated by AI is generated, for example, from an image. The AI ​​generates music using a music generation model. The music generation model is a trained model trained (by machine learning) to generate music based on the relationship between an image and the music to be generated. Therefore, when an image is input, the music generation model generates music corresponding to the image and outputs the generated music. Furthermore, the music generation model may generate music based on performance information input by a user in addition to an image. The performance information input by the user is, for example, information indicating the user's humming or the sound of an instrument being played. The length of the performance is not particularly limited and may be, for example, one phrase. Hereinafter, the second embodiment will be described using an example in which the smart musical instrument 10 outputs, as output information, music generated by AI in accordance with at least one of an image and a performance image input by the user.

[0062] <2-1. Configuration of Musical Piece Creation System> The configuration of a musical piece creation system according to the second embodiment will be described. The musical piece creation system includes a smart musical instrument 10, an imaging device, and a musical piece creation server. Networks used for transmitting and receiving information include, for example, LANs, WANs, telephone networks (such as mobile phone networks and fixed telephone networks), regional IP networks, and the Internet.

[0063] (1) Smart Musical Instrument 10 As described with reference to Fig. 1, the smart musical instrument 10 is a musical instrument having an information processing function and is an example of an information processing device. The smart musical instrument 10 is communicably connected to an imaging device via a wired or wireless connection. The smart musical instrument 10 is also communicably connected to a music generation server via a network.

[0064] A screen for using the music generation system is displayed on the smart musical instrument 10 by an application (hereinafter also referred to as a "music generation app") that allows the user to use the music generation system. The user can generate and play music by operating the screen displayed on the smart musical instrument 10 by the music generation app. The functions of the music generation app may be provided by installing the music generation app on the smart musical instrument 10 (i.e., a native app), or may be provided by a web system (i.e., a web app). In the case of a web app, the music generation app is managed by a server, and its functions are provided via a web browser.

[0065] (2) Imaging Device The imaging device is a device that captures images, such as a camera. The imaging device may be a digital camera or a camera mounted on a smartphone or tablet terminal. The imaging device is communicably connected to the smart musical instrument 10 via a wired or wireless connection. The imaging device transmits captured images to the smart musical instrument 10 in response to user operations.

[0066] (3) Music Generation Server The music generation server is a server that generates music using AI. The music generation server is composed of one or more servers (e.g., cloud servers). For example, the music generation server generates music based on an image in accordance with the content of the image. The music generation server is connected to the smart musical instrument 10 via a network so as to be able to communicate with the smart musical instrument 10.

[0067] <2-2. Functional Configuration of Smart Musical Instrument> The configuration of the music generation system according to the second embodiment has been described above. Next, the functional configuration of the smart musical instrument 10 according to the second embodiment will be described. As described with reference to FIG. 1 , the smart musical instrument 10 includes a communication unit 110, an input unit 120, a storage unit 130, a control unit 140, and an output unit 150.

[0068] (1) Communication Unit 110 The function of the communication unit 110 according to the second embodiment will be described. The communication unit 110 receives captured images in communication with the imaging device. The communication unit 110 also transmits a request to generate music, images, performance information, and the like in communication with the music generation server, and receives music information. The music information includes music generated by the music generation server and information related to the music.

[0069] (2) Input Unit 120 The functions of the input unit 120 according to the second embodiment will be described. The input unit 120 accepts inputs such as a music generation operation, a music playback operation, and a music input operation. A music generation operation is an operation that requests the generation of a music piece. A music playback operation is an operation that requests the playback of a music piece. A music input operation is an operation that inputs performance information that serves as a reference for music generation.

[0070] (3) Storage Unit 130 The following describes the function of the storage unit 130 according to the second embodiment. The storage unit 130 stores captured images, music information, and the like.

[0071] (4) Control Unit 140 The following describes the functions of the control unit 140 according to the second embodiment. As described with reference to FIG. 1 , the control unit 140 includes an acquisition unit 141, an information processing unit 142, and an output processing unit 143.

[0072] (4-1) Acquisition Unit 141 The function of the acquisition unit 141 according to the second embodiment will be described. The acquisition unit 141 acquires an image selected by a user as behavior-induced information. The behavior-induced information is information resulting from the user's behavior of selecting an image. The image (still image or moving image) acquired as behavior-induced information is, for example, a captured image of an arbitrary object (such as a landscape) captured by the user using an imaging device. Note that the image acquired by the acquisition unit 141 is not limited to captured images captured by the user, and may be, for example, captured images captured by a person other than the user, images other than captured images, or images acquired from the Internet, etc. Furthermore, when the user performs a performance input operation, the acquisition unit 141 acquires performance information in addition to the behavior-induced information.

[0073] (4-2) Information Processing Unit 142 The functions of the information processing unit 142 according to the second embodiment will be described. The information processing unit 142 performs processing for generating music, etc. When a user performs a music generation operation, the information processing unit 142 generates music information as output information based on the relationship between an image, performance information, and the music to be generated. For example, the information processing unit 142 generates music information using a music generation model (trained model) that has been trained (by machine learning) to generate music information from the relationship between an image and the music to be generated.

[0074] When only an image is acquired by the acquisition unit 141, the information processing unit 142 generates, as output information, a piece of music corresponding to the content indicated by the acquired image. Specifically, the information processing unit 142 transmits a request for music generation and the image from the communication unit 110 to the music generation server, requesting the generation of music. As a result, the music generation server generates a piece of music corresponding to the image using a music generation model.

[0075] When the acquisition unit 141 acquires the image and performance information, the information processing unit 142 generates, as output information, a piece of music corresponding to the content indicated by the acquired image and performance information. Specifically, the information processing unit 142 transmits a request to generate a piece of music and the image and performance information from the communication unit 110 to the music generation server, requesting the generation of a piece of music. As a result, the music generation server generates a piece of music corresponding to the image and performance information using a music generation model.

[0076] Furthermore, suppose that performance information is input by the user after a piece of music has been generated based only on an image. In this case, the information processing unit 142 generates a first piece of music in accordance with the content indicated by the image, and corrects the first piece of music in accordance with the content indicated by the performance information to generate a second piece of music as output information. Specifically, the information processing unit 142 first transmits a music generation request and an image from the communication unit 110 to the music generation server, requesting the generation of a piece of music. As a result, the music generation server generates a piece of music corresponding to the image using a music generation model. Next, when performance information is input by the user after the generation of the piece of music, the information processing unit 142 transmits a music generation request and practice information from the communication unit 110 to the music generation server, requesting the generation of a piece of music. As a result, the music generation server regenerates (corrects) a piece of music corresponding to the performance information using the music generation model.

[0077] The information processing unit 142 stores the music information received by the communication unit 110 from the music generation server in the storage unit 130. The information processing unit 142 generates output information from the music information. In this way, when an image selected by the user (information resulting from the user's selection) and performance information are input, the information processing unit 142 outputs music generated in accordance with the content indicated by the image and performance information (information corresponding to the user's desire for music that matches the image).

[0078] (4-3) Output Processing Unit 143 The function of the output processing unit 143 according to the second embodiment will be described. The output processing unit 143 performs processing for displaying music information, playing music, and the like. When a user performs a music generation operation, the output processing unit 143 causes the output unit 150 to display information (music information) about the music generated by the music generation server. When a user performs a music playback operation, the output processing unit 143 causes the output unit 150 to output the playback sound of the music selected by the user.

[0079] (5) Output Unit 150 The function of the output unit 150 according to the second embodiment will be described. The output unit 150 displays information about music (music information) on a display device. The output unit 150 also outputs the playback sound of the music from an audio output device.

[0080] <2-3. Functional Configuration of the Music Generation Server> The functional configuration of the smart musical instrument 10 according to the second embodiment has been described above. Next, the functional configuration of the music generation server according to the second embodiment will be described with reference to FIG. 6. FIG. 6 is a block diagram showing an example of the functional configuration of the music generation server according to the second embodiment. As shown in FIG. 6, the music generation server 2030 includes a communication unit 2310, a storage unit 2320, and a control unit 2330.

[0081] (1) Communication Unit 2310 The communication unit 2310 has a function of transmitting and receiving various information. For example, the communication unit 2310 receives a music generation request, an image, performance information, etc. from the smart musical instrument 10 via the network, and transmits music information.

[0082] (2) Storage Unit 2320 The storage unit 2320 has the function of storing various types of information. The storage unit 2320 is configured by a storage medium provided as hardware in the music generation server 2030, such as an HDD, SSD, flash memory, EEPROM, RAM, ROM, or any combination of these storage media. As shown in FIG. 6 , the storage unit 2320 stores a music generation model 2321. The music generation model 2321 is a model (trained model) trained (by machine learning) to generate music based on the relationship between the content indicated by an image and the music to be generated. Therefore, the music generation model 2321 can use an image as input to generate and output music corresponding to the content indicated by the image.

[0083] (3) Control Unit 2330 The control unit 2330 has a function of controlling the overall operation of the music generation server 2030. The control unit 2330 is realized, for example, by causing a CPU or GPU provided as hardware in the music generation server 2030 to execute a program. As shown in FIG. 6 , the control unit 2330 includes a generation training unit 2331 and a music generation unit 2332.

[0084] (3-1) Generation Training Unit 2331 The generation training unit 2331 has a function of training in the generation of music. For example, the generation training unit 2331 trains to generate music from the relationship between the content indicated by the image, the content indicated by the performance information, and the music to be generated, and generates a music generation model 2321 that generates and outputs music corresponding to the content indicated by the input image. The generation training unit 2331 stores the generated music generation model 2321 in the storage unit 2320.

[0085] (3-2) Music Generation Unit 2332 The music generation unit 2332 has a function of generating music information. For example, the music generation unit 2332 generates music information using the music generation model 2321. When the communication unit 2310 receives a music generation request and an image from the smart musical instrument 10, the music generation unit 2332 inputs the image to the music generation model 2321 stored in the storage unit 2320. As a result, the music generation model 2321 generates and outputs a music corresponding to the image. When the communication unit 2310 also receives performance information from the smart musical instrument 10, the music generation unit 2332 inputs the performance information in addition to the image to the music generation model 2321. As a result, the music generation model 2321 generates and outputs a music corresponding to the image and performance information. The music generation unit 2332 acquires the music generated by the music generation model 2321 and generates music information including information about the musical instrument. The music generation unit 2332 transmits the generated music information from the communication unit 2310 to the smart musical instrument 10.

[0086] <2-4. Processing Flow> The functional configuration of the music generation server 2030 according to the second embodiment has been described above. Next, the processing flow according to the second embodiment will be described with reference to FIG. 7. FIG. 7 is a sequence diagram showing an example of the processing flow for generating music according to the second embodiment. Note that the example shown in FIG. 7 describes a case where only an image is input, but processing can also be performed according to a similar flow when performance information is input.

[0087] 7, first, the user performs a photographing operation to photograph an image on the imaging device 2020 (step S2101). The imaging device 2020 photographs an image in response to the user's operation (step S2102). After photographing, the imaging device 2020 transmits the photographed image to the smart musical instrument 10 (step S2103).

[0088] The user performs a music generation operation to generate music on the smart musical instrument 10 (step S2104). When the input unit 120 of the smart musical instrument 10 receives the input of the music generation operation, the information processing unit 142 of the smart musical instrument 10 transmits an image and a generation request from the communication unit 110 to the music generation server 2030 (step S2105).

[0089] When the communication unit 2310 receives the image and the generation request transmitted from the smart musical instrument 10, the music generation unit 2332 of the music generation server 2030 generates music (step S2106). Specifically, the music generation unit 2332 inputs the image into the music generation model 2321 stored in the storage unit 2320 and acquires the music to be output. After generation, the music generation unit 2332 transmits music information including the generated music from the communication unit 2310 to the smart musical instrument 10 (step S2107).

[0090] The output processing unit 143 of the smart musical instrument 10 causes the output unit 150 to display the music information received by the communication unit 110 from the music generation server 2030 (step S2108). The user performs a music playback operation on the smart musical instrument 10 to play a music piece (step S2109). When the input unit 120 accepts the music playback operation input, the information processing unit 142 of the smart musical instrument 10 plays the music piece selected by the user (step S2110).

[0091] As described above, the smart musical instrument 10 according to the second embodiment includes an acquisition unit 141 that acquires at least one of performance information indicating a performance by the user and behavior-induced information that is information different from the performance information and indicates information attributable to the user's behavior, an information processing unit 142 that generates output information indicating non-performance information corresponding to the user's psychology (request) and information related to music based on the acquired information of the performance information and the behavior-induced information, and an output processing unit 143 that outputs the output information. The acquisition unit 141 acquires an image selected by the user as behavior-induced information and also acquires performance information, and the information processing unit 142 generates a song as output information corresponding to the content indicated by the acquired image and performance information.

[0092] With this configuration, even when the user is not playing a musical instrument in daily life, the user can input information resulting from the action of selecting a target image into the smart musical instrument 10, and obtain information from the smart musical instrument 10 that corresponds to the user's psychology of wanting a piece of music that matches the image. This allows the user to feel that the smart musical instrument 10 is supporting the user when creating a piece of music. Therefore, the smart musical instrument 10 according to the second embodiment can enhance the sense of symbiosis with music.

[0093] <<3. Third Embodiment>> A third embodiment will be described with reference to FIGS. 8 and 9 . In the third embodiment, a communication system using a smart musical instrument 10 will be described. The communication system in the third embodiment is a system using AI, in which the AI ​​communicates with a user. In the third embodiment, communication between the user and the AI ​​is performed via the smart musical instrument 10. Communication from the user may be, for example, playing the smart musical instrument 10, or may be a means other than playing, such as touching or talking to the smart musical instrument 10. The smart musical instrument 10 outputs a reaction to communication from the user (e.g., acting spoiled, sulking, etc.). The AI ​​in the third embodiment generates information indicating a reaction in response to communication from the user (hereinafter also referred to as "reaction information"). The reaction information is generated by the AI ​​using a communication model. The communication model is a trained model trained (by machine learning) to generate reaction information based on the relationship between communication from the user and the generated reaction information. For this reason, when information indicating communication from a user (hereinafter also referred to as "input information") is input to the communication model, the communication model generates reaction information indicating a reaction (communication) corresponding to the input information and outputs the generated reaction information. Below, a third embodiment will be described using an example in which the smart musical instrument 10 outputs, as output information, reaction information generated by AI in response to input information input from the user. In addition, in the third embodiment, an example in which the reaction (communication) of the smart musical instrument 10 is output via an avatar that personifies the smart musical instrument 10 will be described.

[0094] <3-1. Configuration of the Communication System> The configuration of the communication system according to the third embodiment will be described. The communication system includes a smart musical instrument 10 and a communication server. The network used for transmitting and receiving information may be, for example, a LAN, a WAN, a telephone network (such as a mobile phone network or a fixed telephone network), a regional IP network, or the Internet.

[0095] (1) Smart Musical Instrument 10 As described with reference to Fig. 1, the smart musical instrument 10 is a musical instrument having an information processing function and is an example of an information processing device. The smart musical instrument 10 is communicably connected to a communication server via a network.

[0096] A screen for using the communication system is displayed on the smart musical instrument 10 by an application (hereinafter also referred to as a "communication app") that allows the user to use the communication system. The user can communicate with the smart musical instrument 10 by operating the screen displayed on the smart musical instrument 10 by the communication app. The functions of the communication app may be provided by installing the communication app on the smart musical instrument 10 (i.e., a native app), or may be provided by a web system (i.e., a web app). In the case of a web app, the communication app is managed by a server, and its functions are provided via a web browser.

[0097] (2) Communication Server The communication server is a server that generates reaction information to user communication using AI. The communication server is composed of one or more servers (e.g., cloud servers). For example, the communication server generates reaction information based on input information in accordance with the communication content indicated by the input information. The communication server is communicably connected to the smart musical instrument 10 via a network.

[0098] <3-2. Functional Configuration of Smart Musical Instrument> The configuration of the communication system according to the third embodiment has been described above. Next, the functional configuration of the smart musical instrument 10 according to the third embodiment will be described. As described with reference to FIG. 1 , the smart musical instrument 10 includes a communication unit 110, an input unit 120, a storage unit 130, a control unit 140, and an output unit 150.

[0099] (1) Communication Unit 110 The following describes the functions of the communication unit 110 according to the third embodiment. The communication unit 110 transmits a reaction information generation request and input information and receives reaction information in communication with the communication server.

[0100] (2) Input Unit 120 The function of the input unit 120 according to the third embodiment will be described. The input unit 120 accepts communication input from a user. The communication input from a user corresponds to an operation requesting the generation of reaction information.

[0101] (3) Storage Unit 130 The function of the storage unit 130 according to the third embodiment will be described below. The storage unit 130 stores avatar information and the like.

[0102] (4) Control Unit 140 The following describes the functions of the control unit 140 according to the third embodiment. As described with reference to FIG. 1 , the control unit 140 includes an acquisition unit 141, an information processing unit 142, and an output processing unit 143.

[0103] (4-1) Acquisition Unit 141 The function of the acquisition unit 141 according to the third embodiment will be described. The acquisition unit 141 acquires input information indicating communication from the user as behavior-induced information. The behavior-induced information is information resulting from the behavior of communication. When the user communicates with the smart musical instrument 10 by a method other than playing an instrument (not limited to the smart musical instrument 10), the acquisition unit 141 acquires information indicating the communication as behavior-induced information. Examples of communication other than playing an instrument include touching the smart musical instrument 10, talking to the smart musical instrument 10, and moving the smart musical instrument 10 to a predetermined location. Note that when the user communicates with the smart musical instrument 10 by playing an instrument (not limited to the smart musical instrument 10), the acquisition unit 141 acquires performance information indicating the content of the performance.

[0104] (4-2) Information Processing Unit 142 The function of the information processing unit 142 according to the third embodiment will be described. The information processing unit 142 performs processing for responding to communication from the user. When communication is received from the user, the information processing unit 142 generates, as output information, reaction information indicating the reaction of the smart musical instrument 10 to the communication from the user, based on at least one of the performance information and the behavior-induced information acquired by the acquisition unit 141. For example, the information processing unit 142 generates the reaction information using a communication model (trained model) that has been trained (by machine learning) to generate reaction information based on the relationship between the communication from the user and the generated reaction information. Specifically, the information processing unit 142 transmits input information from the communication unit 110 to the communication server and requests a reaction to the communication indicated by the input information. As a result, the communication server generates reaction information corresponding to the input information using the communication model.

[0105] The information processing unit 142 stores the reaction information received by the communication unit 110 from the communication server in the storage unit 130. The information processing unit 142 generates output information from the reaction information. In this way, when communication (information resulting from the user's behavior of communication) is input from the user, the information processing unit 142 generates and outputs a reaction corresponding to the communication (information corresponding to the user's desire for a reaction from the instrument).

[0106] (4-3) Output Processing Unit 143 The function of the output processing unit 143 according to the third embodiment will be described. The output processing unit 143 performs processing for displaying reaction information, which is output information, and outputting audio. The output processing unit 143 outputs the output information via an avatar that personifies the smart musical instrument 10. For example, the output processing unit 143 displays the output information together with the avatar on a display device (output unit 150) such as a monitor. As an example, the output processing unit 143 displays text or images (still images or videos) indicating the content of the output information together with the avatar. The output processing unit 143 may also display the avatar on the display device and then output the output information as audio from an audio output device (output unit 150) such as a speaker. As an example, the output processing unit 143 outputs audio indicating the content of the output information as if the avatar were speaking. The output processing unit 143 may also have the avatar explain the content of the output information displayed on the display device. In this case, the output processing unit 143 may display text indicating the content of the explanation by the avatar on the display device, or may output audio indicating the content of the explanation by the avatar from the audio output device. The output processing unit 143 may also output the sound of the avatar playing the smart musical instrument 10 from the audio output device. For example, when the reaction information generated by the information processing unit 142 is a performance by the avatar of the smart musical instrument 10, the output processing unit 143 outputs the performance sound indicated by the reaction information as output information from the audio output device. The output processing unit 143 may also output audio corresponding to the content of the output information from the audio output device and display an avatar performing an action corresponding to the content of the output information on the display device. For example, when the reaction information generated by the information processing unit 142 is a performance by the avatar of the smart musical instrument 10, the output processing unit 143 outputs the sound of the avatar playing the smart musical instrument 10 from the audio output device and display an avatar performing an action corresponding to the performance sound on the display device. The output indicating the action of the avatar (avatar animation) is generated, for example, using XR (cross reality) technology.

[0107] (5) Output Unit 150 The function of the output unit 150 according to the third embodiment will be described. The output unit 150 displays an image indicating the reaction information on a display device. The output unit 150 also outputs a sound indicating the reaction information from an audio output device.

[0108] <3-3. Functional Configuration of the Communication Server> The functional configuration of the smart musical instrument 10 according to the third embodiment has been described above. Next, the functional configuration of the communication server according to the third embodiment will be described with reference to FIG. 8. FIG. 8 is a block diagram showing an example of the functional configuration of the communication server according to the third embodiment. As shown in FIG. 8, the communication server 3020 includes a communication unit 3210, a storage unit 3220, and a control unit 3230.

[0109] (1) Communication Unit 3210 The communication unit 3210 has a function of transmitting and receiving various information. For example, the communication unit 3210 receives input information from the smart musical instrument 10 and transmits response information via the network.

[0110] (2) Storage Unit 3220 The storage unit 3220 has a function of storing various information. The storage unit 3220 is configured by a storage medium provided as hardware in the communication server 3020, such as an HDD, SSD, flash memory, EEPROM, RAM, ROM, or any combination of these storage media. As shown in FIG. 8 , the storage unit 3220 stores a communication model 3221. The communication model 3221 is a model (trained model) trained (by machine learning) to generate reaction information based on the relationship between communication from a user and the reaction information to be generated. Therefore, the communication model 3221 can receive input information as input and generate and output reaction information corresponding to the communication indicated by the input information.

[0111] (3) Control Unit 3230 The control unit 3230 has a function of controlling the overall operation of the communication server 3020. The control unit 3230 is realized, for example, by causing a CPU or GPU provided as hardware in the communication server 3020 to execute a program. As shown in FIG. 8 , the control unit 3230 includes a communication training unit 3231 and a communication generation unit 3232.

[0112] (3-1) Communication Training Unit 3231 The communication training unit 3231 has a function of training the generation of communication (reaction information). For example, the communication training unit 3231 trains to generate reaction information from the relationship between the content of communication from the user and the content of communication to be generated, and generates a communication model 3221 that generates and outputs reaction information corresponding to input information. The communication training unit 3231 stores the generated communication model 3221 in the storage unit 3220.

[0113] (3-2) Communication Generation Unit 3232 The communication generation unit 3232 has a function of generating communication (reaction information). For example, the communication generation unit 3232 generates reaction information using the communication model 3221. When the communication unit 3210 receives input information from the smart musical instrument 10, the communication generation unit 3232 inputs the input information to the communication model 3221 stored in the storage unit 3220. As a result, reaction information corresponding to the input information is generated and output by the communication model 3221. The communication generation unit 3232 transmits the reaction information generated by the communication model 3221 from the communication unit 3210 to the smart musical instrument 10.

[0114] <3-4. Processing Flow> The functional configuration of the communication server 3020 according to the third embodiment has been described above. Next, the processing flow according to the third embodiment will be described with reference to Fig. 9. Fig. 9 is a sequence diagram showing an example of the flow of communication processing according to the third embodiment.

[0115] 9 , first, the user inputs input information by communicating with the smart musical instrument 10 (step S3101). When the input unit 120 of the information processing unit 142 of the smart musical instrument 10 receives the input information, the information processing unit 142 transmits the input information from the communication unit 110 to the communication server 3020 (step S3102).

[0116] When the communication unit 3210 receives input information transmitted from the smart musical instrument 10, the communication generation unit 3232 of the communication server 3020 generates reaction information (step S3103). Specifically, the communication generation unit 3232 inputs the input information to the communication model 3221 stored in the storage unit 3220 and acquires the reaction information to be output. After generation, the communication generation unit 3232 transmits the generated reaction information from the communication unit 3210 to the smart musical instrument 10 (step S3104).

[0117] The output processing unit 143 of the smart musical instrument 10 causes the output unit 150 to output the reaction information received by the communication unit 110 from the communication server 3020 (step S3105). The user checks the reaction of the smart musical instrument 10 to the user's communication, which is output from the smart musical instrument 10 (step S3106).

[0118] As described above, the smart musical instrument 10 according to the third embodiment includes an acquisition unit 141 that acquires at least one of performance information indicating a performance by the user and behavior-induced information that is information different from the performance information and indicates information attributable to the user's behavior; an information processing unit 142 that generates output information indicating non-performance information that corresponds to the user's psychology (request) and information related to music based on the acquired performance information or behavior-induced information; and an output processing unit 143 that outputs the output information. The acquisition unit 141 acquires information indicating the user's communication with the musical instrument as performance information in the case of communication by playing the musical instrument, and as behavior-induced information in the case of communication by means other than playing the musical instrument. The information processing unit 142 generates information indicating the musical instrument's response to the communication from the user as output information based on the acquired performance information or behavior-induced information.

[0119] With this configuration, even when the user is not playing a musical instrument in daily life, the user can input information resulting from the act of communication to the smart musical instrument 10, thereby obtaining information from the smart musical instrument 10 that corresponds to the user's desire for a response from the smart musical instrument 10. This allows the user to feel that the smart musical instrument 10 is there for the user when communicating with the smart musical instrument 10. Therefore, the smart musical instrument 10 according to the third embodiment can enhance the sense of symbiosis with music.

[0120] <<4. Fourth Embodiment>> A fourth embodiment will be described with reference to FIGS. 10 and 11 . In the fourth embodiment, an avatar management system using a smart musical instrument 10 will be described. The avatar management system in the fourth embodiment is a system using AI, in which the AI ​​manages avatars. In the third embodiment described above, a communication system was described in which a user communicates with the smart musical instrument 10, allowing the smart musical instrument 10 to communicate with the smart musical instrument 10 via an anthropomorphized avatar. In the fourth embodiment, the avatar management system manages avatars used in the communication system, and the AI ​​grows the avatars. In the fourth embodiment, the AI ​​grows the avatars according to the experience of the smart musical instrument 10. The experience of the smart musical instrument 10 is determined based on information input to the smart musical instrument 10 (hereinafter also referred to as “input information”). The input information may be, for example, location information of places where the user has taken the smart musical instrument 10 (places where the user has moved the smart musical instrument 10) or performance information indicating the content of the user's performance of an instrument (not limited to the smart musical instrument 10). The AI ​​generates information indicating the growth level of the smart musical instrument 10 (hereinafter also referred to as "growth information") based on the input information. The generation of growth information by the AI ​​is performed using an avatar management model. The avatar management model is a trained model trained (by machine learning) to generate growth information based on the relationship between the content indicated by the input information and the growth level. Therefore, when input information is input to the smart musical instrument 10, the avatar management model grows the avatar according to the input information, and generates and outputs growth information. Below, the fourth embodiment will be described using an example in which the smart musical instrument 10 outputs growth information generated by the AI ​​according to the input information input to the smart musical instrument 10 as output information.

[0121] <4-1. Configuration of the Avatar Management System> The configuration of the avatar management system according to the fourth embodiment will be described. The avatar management system includes a smart musical instrument 10, a user terminal, and an avatar management server. Networks used for transmitting and receiving information include, for example, LANs, WANs, telephone networks (such as mobile phone networks and fixed telephone networks), regional IP networks, and the Internet.

[0122] (1) Smart Musical Instrument 10 As described with reference to Fig. 1, the smart musical instrument 10 is a musical instrument having an information processing function and is an example of an information processing device. The smart musical instrument 10 is communicably connected to a user terminal and an avatar management server via a network.

[0123] A screen for using the avatar management system is displayed on the smart musical instrument 10 by an application (hereinafter also referred to as an "avatar management app") that allows the user to use the avatar management system. The user can manage the avatars displayed via the smart musical instrument 10 by operating the screen displayed on the smart musical instrument 10 by the avatar management app. Note that the functions of the avatar management app may be provided by installing the avatar management app on the smart musical instrument 10 (i.e., a native app), or may be provided by a web system (i.e., a web app). In the case of a web app, the avatar management app is managed by a server, and its functions are provided via a web browser.

[0124] (2) User Terminal The user terminal is a terminal owned by a user who owns the smart musical instrument 10. The user terminal is, for example, a smartphone, a tablet terminal, a wearable terminal, or a PC. The user terminal is connected to the smart musical instrument 10 via a network so as to be able to communicate with the smart musical instrument 10.

[0125] In the fourth embodiment, the user terminal is used as a terminal for acquiring location information of the smart musical instrument 10. Therefore, the user terminal is a terminal capable of acquiring location information. If the smart musical instrument 10 has a function for acquiring location information by itself, a user terminal may not be used.

[0126] (3) Avatar Management Server The avatar management server is a server that manages avatars using AI. The avatar management server is composed of one or more servers (e.g., cloud servers). The avatar management server manages the growth of avatars based on, for example, input information. The avatar management server is connected to the smart musical instrument 10 via a network so as to be able to communicate with the smart musical instrument 10.

[0127] <4-2. Functional Configuration of Smart Musical Instrument> The configuration of the avatar management system according to the fourth embodiment has been described above. Next, the functional configuration of the smart musical instrument 10 according to the fourth embodiment will be described. As described with reference to FIG. 1 , the smart musical instrument 10 includes a communication unit 110, an input unit 120, a storage unit 130, a control unit 140, and an output unit 150.

[0128] (1) Communication Unit 110 The function of the communication unit 110 according to the fourth embodiment will be described. The communication unit 110 receives location information in communication with a user terminal. The communication unit 110 also transmits input information and receives growth information in communication with an avatar management server.

[0129] (2) Input Unit 120 The function of the input unit 120 according to the fourth embodiment will be described below. The input unit 120 receives input of performance information from the user.

[0130] (3) Storage Unit 130 The function of the storage unit 130 according to the fourth embodiment will be described. The storage unit 130 stores growth information and the like.

[0131] (4) Control Unit 140 The following describes the functions of the control unit 140 according to the fourth embodiment. As described with reference to FIG. 1 , the control unit 140 includes an acquisition unit 141, an information processing unit 142, and an output processing unit 143.

[0132] (4-1) Acquisition Unit 141 The function of the acquisition unit 141 according to the fourth embodiment will be described. The acquisition unit 141 acquires position information of an avatar that personifies the smart musical instrument 10, which changes as the user moves the smart musical instrument 10. The acquisition unit 141 also acquires performance information that indicates the performance content of the user using the musical instrument. The position information is behavior-induced information resulting from the behavior of movement.

[0133] (4-2) Information Processing Unit 142 The function of the information processing unit 142 according to the fourth embodiment will be described. The information processing unit 142 performs processing to develop the smart musical instrument 10 based on input information. When location information is acquired, the information processing unit 142 develops the avatar according to the location indicated by the acquired location information and generates, as output information, development information indicating the degree of development. When performance information is acquired, the information processing unit 142 develops the avatar based on the acquired performance information and generates, as output information, development information indicating the degree of development. For example, if the location indicated by the location information is a music-related venue such as a concert venue or live music club, the information processing unit 142 develops the avatar. Furthermore, the information processing unit 142 may develop the avatar according to the genre of music played at that venue. The avatar's development may involve, for example, increasing the avatar's knowledge of the genre or learning a performance style. For example, if location information of live music clubs where rock bands often perform is acquired frequently, the information processing unit 142 develops the avatar into one that is knowledgeable about rock music or that can play rock music. In this case, the information processing unit 142 generates, for example, information indicating the level of proficiency in each genre (such as knowledge level and performance ability) as growth information. For example, the information processing unit 142 uses an avatar management model (trained model) that has been trained (by machine learning) to generate growth information based on the relationship between the content indicated by the input information and the level of growth to grow the avatar and generate the growth information. Specifically, the information processing unit 142 transmits the input information from the communication unit 110 to the avatar management server and requests that the avatar be grown according to the content indicated by the input information. As a result, the avatar management server uses the avatar management model to grow the avatar according to the input information and generate the growth information.

[0134] The information processing unit 142 may also generate output information based on the generated growth information. For example, the information processing unit 142 generates output information according to the degree of growth of the avatar indicated by the growth information. As an example, the information processing unit 142 generates performance sounds (e.g., accompaniment sounds, music, etc.) based on the performance style of the avatar as output information.

[0135] The information processing unit 142 stores the growth information received by the communication unit 110 from the avatar management server in the storage unit 130. The information processing unit 142 generates output information from the growth information. In this way, when input information (information resulting from the user's action of moving) is input to the smart musical instrument 10, the information processing unit 142 generates and outputs growth information corresponding to the input information (information corresponding to the user's desire to grow the musical instrument).

[0136] (4-3) Output Processing Unit 143 The function of the output processing unit 143 according to the fourth embodiment will be described. The output processing unit 143 performs processing for outputting growth information, which is output information. For example, the output processing unit 143 displays the growth information along with the avatar on a display device (output unit 150), such as a display. The output processing unit 143 may also output information indicating the reaction (communication) of the grown avatar from the display device or audio output device. The output processing unit 143 may also output the performance sound of the smart musical instrument 10 by the avatar as output information. For example, the output processing unit 143 outputs the performance sound of the smart musical instrument 10 generated by the information processing unit 142 according to the avatar's level of growth (e.g., playing style and its proficiency) from the audio output device. At this time, the output processing unit 143 may display on the display device an avatar performing a motion in accordance with the output performance sound. As an example, the output processing unit 143 displays on the display device an avatar performing a performance motion according to the avatar's playing style and proficiency.

[0137] (5) Output Unit 150 The following describes the function of the output unit 150 according to the fourth embodiment. The output unit 150 outputs information indicating growth information and avatar communication.

[0138] <4-3. Functional Configuration of the Avatar Management Server> The functional configuration of the smart musical instrument 10 according to the fourth embodiment has been described above. Next, the functional configuration of the avatar management server according to the fourth embodiment will be described with reference to FIG. 10. FIG. 10 is a block diagram showing an example of the functional configuration of the avatar management server according to the fourth embodiment. As shown in FIG. 10, the avatar management server 4030 includes a communication unit 4310, a storage unit 4320, and a control unit 4330.

[0139] (1) Communication Unit 4310 The communication unit 4310 has a function of transmitting and receiving various information. For example, the communication unit 4310 receives input information from the smart musical instrument 10 and transmits growth information via the network.

[0140] (2) Storage Unit 4320 The storage unit 4320 has the function of storing various types of information. The storage unit 4320 is configured by a storage medium provided as hardware in the avatar management server 4030, such as an HDD, SSD, flash memory, EEPROM, RAM, ROM, or any combination of these storage media. As shown in FIG. 10 , the storage unit 4320 stores an avatar management model 4321. The avatar management model 4321 is a model (trained model) trained (by machine learning) to generate growth information based on the relationship between the content indicated by input information and the growth level. Therefore, the avatar management model 4321 receives input information, grows an avatar according to the content indicated by the input information, and generates and outputs growth information.

[0141] (3) Control Unit 4330 The control unit 4330 has a function of controlling the overall operation of the avatar management server 4030. The control unit 4330 is realized, for example, by causing a CPU or GPU provided as hardware in the avatar management server 4030 to execute a program. As shown in FIG. 10 , the control unit 4330 includes a management training unit 4331 and an avatar management unit 4332.

[0142] (3-1) Management Training Unit 4331 The management training unit 4331 has a function of training the growth of an avatar. For example, the management training unit 4331 trains the unit 4331 to generate growth information based on the relationship between the content indicated by input information and the growth level of the avatar, grows the avatar according to the input information, and generates an avatar management model 4321 that generates and outputs growth information. The management training unit 4331 stores the generated avatar management model 4321 in the storage unit 4320.

[0143] (3-2) Avatar Management Unit 4332 The avatar management unit 4332 has a function of growing an avatar and generating growth information. For example, the avatar management unit 4332 grows an avatar and generates growth information using the avatar management model 4321. When the communication unit 4310 receives input information from the smart musical instrument 10, the avatar management unit 4332 inputs the input information to the avatar management model 4321 stored in the storage unit 4320. As a result, the avatar management model 4321 grows the avatar and generates reaction information according to the input information, and outputs the growth information. The avatar management unit 4332 transmits the growth information generated by the avatar management model 4321 from the communication unit 4310 to the smart musical instrument 10.

[0144] <4-4. Processing Flow> The functional configuration of the avatar management server 4030 according to the fourth embodiment has been described above. Next, the processing flow according to the fourth embodiment will be described with reference to Fig. 11. Fig. 11 is a sequence diagram showing an example of the flow of avatar management processing according to the fourth embodiment.

[0145] 11 , first, the user terminal 4020 acquires location information (step S4101). After acquiring the location information, the user terminal 4020 transmits the acquired location information to the smart musical instrument 10 (step S4102). The user plays the smart musical instrument 10 (step S4103).

[0146] The information processing unit 142 of the smart musical instrument 10 transmits at least one of the position information received from the user terminal 4020 and the performance information input by the user as input information from the communication unit 110 to the avatar management server 4030 (step S4104).

[0147] When the communication unit 4310 receives the input information transmitted from the smart musical instrument 10, the avatar management unit 4332 of the avatar management server 4030 performs a process of growing the avatar (step S4105). Specifically, the avatar management unit 4332 inputs the input information into the avatar management model 4321 stored in the storage unit 4320, grows the avatar, and obtains growth information to be output. After the growth process, the avatar management unit 4332 transmits the growth information from the communication unit 4310 to the smart musical instrument 10 (step S4106).

[0148] The output processing unit 143 of the smart musical instrument 10 displays the growth information received by the communication unit 110 from the avatar management server 4030 on the output unit 150 (step S4107). The user can check the growth level of the avatar from the growth information displayed on the smart musical instrument 10 (step S4108).

[0149] As described above, the smart musical instrument 10 according to the fourth embodiment includes an acquisition unit 141 that acquires at least one of performance information indicating a performance by the user and behavior-induced information that is information different from the performance information and indicates information attributable to the user's behavior; an information processing unit 142 that generates output information indicating non-performance information corresponding to the user's psychology (request) based on the acquired information of the performance information and the behavior-induced information and that indicates information related to music; and an output processing unit 143 that outputs the output information. The acquisition unit 141 acquires, as behavior-induced information, position information of an avatar that anthropomorphizes the musical instrument, which position information changes as the user moves the instrument. The information processing unit 142 grows the avatar according to the location indicated by the acquired position information and generates, as output information, information indicating communication content corresponding to the growth.

[0150] With this configuration, the user can input information resulting from the action of taking the smart musical instrument 10 out, even when not playing the instrument in daily life, to the smart musical instrument 10, thereby obtaining information from the smart musical instrument 10 that corresponds to the user's desire to develop the smart musical instrument 10. As a result, as the smart musical instrument 10 develops, the user can obtain information that is more suited to the user from the smart musical instrument 10, and can feel that the smart musical instrument 10 is supporting the user. Therefore, the smart musical instrument 10 according to the fourth embodiment can improve the sense of coexistence with music.

[0151] <<5. Fifth Embodiment>> A fifth embodiment will be described with reference to FIG. 12 . In the fifth embodiment, a musical instrument management system using a smart musical instrument 10 will be described. The musical instrument management system is a system that can accumulate various information acquired after a user starts using the smart musical instrument 10 and output the information as needed. The fifth embodiment will be described below using as an example an example in which the smart musical instrument 10 acquires history information corresponding to trigger information input to the smart musical instrument 10 and outputs the history information as output information.

[0152] 5-1. Configuration of the musical instrument management system The following describes the configuration of the musical instrument management system according to the fifth embodiment. The musical instrument management system includes a smart musical instrument 10 and a user terminal.

[0153] (1) Smart Musical Instrument 10 As described with reference to Fig. 1, the smart musical instrument 10 is a musical instrument having an information processing function and is an example of an information processing device. The smart musical instrument 10 is connected to a user terminal via a wired or wireless connection so as to be able to communicate with the user terminal.

[0154] A screen for using the instrument management system is displayed on the smart instrument 10 by an application (hereinafter also referred to as an "instrument management app") that allows the user to use the instrument management system. The user can manage various information stored in the smart instrument 10 by operating the screen displayed on the smart instrument 10 by the instrument management app. The functions of the instrument management app may be provided by installing the instrument management app on the smart instrument 10 (i.e., a native app), or may be provided by a web system (i.e., a web app). In the case of a web app, the instrument management app is managed by a server, and its functions are provided via a web browser.

[0155] (2) User Terminal The user terminal is a terminal owned by a user who owns the smart musical instrument 10. The user terminal may be, for example, a smartphone, a tablet terminal, a wearable terminal, or a PC. The user terminal is connected to the smart musical instrument 10 via a wired or wireless connection so as to be able to communicate with the smart musical instrument 10.

[0156] <5-2. Functional Configuration of Smart Musical Instrument> The configuration of the musical instrument management system according to the fifth embodiment has been described above. Next, the functional configuration of the smart musical instrument 10 according to the fifth embodiment will be described. As described with reference to FIG. 1 , the smart musical instrument 10 includes a communication unit 110, an input unit 120, a storage unit 130, a control unit 140, and an output unit 150.

[0157] (1) Communication Unit 110 The function of the communication unit 110 according to the fifth embodiment will be described. The communication unit 110 receives trigger information through communication with a user terminal. The trigger information transmitted from the user terminal is, for example, location information. The location information is behavior-induced information resulting from the user's behavior of moving.

[0158] (2) Input Unit 120 The function of the input unit 120 according to the fifth embodiment will be described. The input unit 120 receives input of trigger information from a user. The trigger information from the user is, for example, information indicating that performance information or information to be output has been selected.

[0159] (3) Storage Unit 130 The function of the storage unit 130 according to the fifth embodiment will be described. The storage unit 130 stores history information. The history information is performance information and behavior-induced information acquired by the acquisition unit 141 after the user starts using the musical instrument.

[0160] (4) Control Unit 140 The following describes the functions of the control unit 140 according to the fifth embodiment. As described with reference to FIG. 1 , the control unit 140 includes an acquisition unit 141, an information processing unit 142, and an output processing unit 143.

[0161] (4-1) Acquisition Unit 141 The function of the acquisition unit 141 according to the fifth embodiment will be described below. The acquisition unit 141 acquires trigger information that serves as a trigger for outputting output information.

[0162] (4-2) Information Processing Unit 142 The function of the information processing unit 142 according to the fifth embodiment will be described. The information processing unit 142 generates output information from history information stored in the storage unit 130 based on acquired trigger information. Specifically, the information processing unit 142 refers to the storage unit 130 and acquires history information corresponding to the trigger information. For example, if the trigger information is location information, the information processing unit 142 acquires history information stored at the location indicated by the location information. Furthermore, if the trigger information is performance information, the information processing unit 142 acquires history information related to the performance content indicated by the performance information.

[0163] (4-3) Output Processing Unit 143 The following describes the function of the output processing unit 143 according to the fifth embodiment. The output processing unit 143 causes the output unit 150 to output output information generated from history information.

[0164] (5) Output Unit 150 The function of the output unit 150 according to the fifth embodiment will be described below. The output unit 150 outputs output information generated from history information.

[0165] <5-3. Processing Flow> The functional configuration of the smart musical instrument 10 according to the fifth embodiment has been described above. Next, the processing flow according to the fifth embodiment will be described with reference to Fig. 12. Fig. 12 is a sequence diagram showing an example of the flow of a process for outputting past information according to the fifth embodiment.

[0166] 12 , the user inputs trigger information to the smart musical instrument 10 (step S5101). The information processing unit 142 of the smart musical instrument 10 acquires history information corresponding to the trigger information received from the smart musical instrument 10 from the storage unit 130 (step S5102). After acquisition, the information processing unit 142 generates output information from the acquired history information (step S5103). After generation, the output processing unit 143 outputs the output information (step S5104). The user checks the output information output from the smart musical instrument 10 (step S5105).

[0167] As described above, the smart musical instrument 10 according to the fifth embodiment includes an acquisition unit 141 that acquires at least one of performance information indicating a performance by the user and behavior-induced information that is information different from the performance information and indicates information attributable to the user's behavior; an information processing unit 142 that generates output information indicating non-performance information that corresponds to the user's psychology (request) based on the acquired information of the performance information and the behavior-induced information, and information related to music; and an output processing unit 143 that outputs the output information. The smart musical instrument 10 further includes a memory unit 130 that stores the performance information and behavior-induced information acquired by the acquisition unit 141 as history information after the user starts using the instrument. The acquisition unit 141 acquires trigger information that serves as a trigger for outputting the output information, and the information processing unit 142 generates output information from the history information stored in the memory unit 130 based on the acquired trigger information.

[0168] With this configuration, even when the user is not playing a musical instrument in daily life, the user can input trigger information, which is information resulting from the user's actions, to the smart musical instrument 10, and obtain information from the smart musical instrument 10 that reflects the user's desire to reminisce about the past. This allows the user to enjoy memories with the smart musical instrument 10 and feel that the smart musical instrument 10 is there for them. Therefore, the smart musical instrument 10 according to the fifth embodiment can enhance the sense of coexistence with music.

[0169] The trigger information may be information indicating a change in the user of the smart musical instrument 10 (hereinafter, also referred to as "user change information"). The user change information is acquired, for example, when a different user plays (operates) the smart musical instrument 10, resulting in a change in the performer, or when the smart musical instrument 10 is handed over to another user and the owner changes. When the acquisition unit 141 acquires the user change information as trigger information, output information generated from the history information of the previous user (past user) may be output. In this case, the information processing unit 142 refers to the storage unit 130 to acquire history information corresponding to the user change information. For example, the information processing unit 142 acquires the history information of the past user and generates output information related to the past user. The output processing unit 143 outputs the output information generated from the history information of the past user to the output unit 150. When the acquisition unit 141 acquires the user change information as trigger information, performance information acquired from the history information of the past user may be output. In this case, the information processing unit 142 acquires the performance information of the past user from the history information stored in the storage unit 130 based on the trigger information. The output processing unit 143 outputs the performance information of the past user acquired by the information processing unit 142. As an example, the output processing unit 143 plays back the past performance of the past user based on the performance information of the past user and outputs it to the output unit 150. This allows the changed user (the current user) to experience as if they were interacting with the past user through music. Furthermore, the output processing unit 143 may output both the past performance of the past user and the current performance of the current user to the output unit 150. This allows the current user to have a session with the past user.

[0170] <<6. Sixth Embodiment>> A sixth embodiment will be described with reference to FIG. 13 . In the sixth embodiment, a practice assistance system using a smart musical instrument 10 will be described. The practice assistance system is a system that assists in sports practice. For example, the practice assistance system converts exercise information indicating a user's movements acquired while the user is practicing a sport into audio and simultaneously outputs the audio together with audio indicating a model movement for comparison. By comparing the simultaneously output audio, the user can recognize discrepancies between their own movements and the model movement and correct their movements. Hereinafter, the sixth embodiment will be described using an example in which the smart musical instrument 10 generates audio corresponding to exercise information input to the smart musical instrument 10 and outputs the audio as output information together with the model audio.

[0171] 6-1. Configuration of Practice Support System The following describes the configuration of a practice support system according to the sixth embodiment. The practice support system includes a smart musical instrument 10, a sensor device, and an audio output device.

[0172] (1) Smart Musical Instrument 10 As described with reference to Fig. 1, the smart musical instrument 10 is a musical instrument having an information processing function and is an example of an information processing device. The smart musical instrument 10 is communicably connected to a sensor device and an audio output device via a wired or wireless connection.

[0173] A screen for using the practice assistance system is displayed on the smart musical instrument 10 by an application (hereinafter also referred to as a "practice assistance app") that allows the user to use the practice assistance system. The user can receive practice assistance by operating the screen displayed on the smart musical instrument 10 by the practice assistance app. The functions of the practice assistance app may be provided by installing the practice assistance app on the smart musical instrument 10 (i.e., a native app), or may be provided by a web system (i.e., a web app). In the case of a web app, the practice assistance app is managed by a server, and its functions are provided via a web browser.

[0174] (2) Sensor Device The sensor device acquires motion information indicating the user's movements during practice. For example, the sensor device may be an acceleration sensor, and acquires information indicating acceleration that changes with the user's movements as the motion information. The sensor device is connected to the smart musical instrument 10 via a wired or wireless connection so as to be able to communicate with the smart musical instrument 10.

[0175] (3) Audio Output Device The audio output device is a device that outputs audio. The audio output device is preferably a device, such as earphones or headphones, that can output audio corresponding to the user's actions or audio corresponding to a model action to the user's left and right ears. The audio output device is connected to the smart musical instrument 10 via a wired or wireless connection so as to be able to communicate with the smart musical instrument 10.

[0176] <6-2. Functional Configuration of Smart Musical Instrument> The configuration of the practice assistance system according to the sixth embodiment has been described above. Next, the functional configuration of the smart musical instrument 10 according to the sixth embodiment will be described. As described with reference to FIG. 1 , the smart musical instrument 10 includes a communication unit 110, an input unit 120, a storage unit 130, a control unit 140, and an output unit 150.

[0177] (1) Communication Unit 110 The function of the communication unit 110 according to the sixth embodiment will be described. The communication unit 110 receives exercise information through communication with the sensor device. Furthermore, the communication unit 110 transmits audio corresponding to the user's movements and audio corresponding to the model movements through communication with the audio output device.

[0178] (2) Input Unit 120 The function of the input unit 120 according to the sixth embodiment will be described below. The input unit 120 receives communication inputs that receive various operations from the user.

[0179] (3) Storage Unit 130 The function of the storage unit 130 according to the sixth embodiment will be described. The storage unit 130 stores history information and model information. The storage unit 130 stores, in advance, as model information, audio that indicates a change in a motion that serves as a model for the motion performed by the user.

[0180] (4) Control Unit 140 The following describes the functions of the control unit 140 according to the sixth embodiment. As described with reference to FIG. 1 , the control unit 140 includes an acquisition unit 141, an information processing unit 142, and an output processing unit 143.

[0181] (4-1) Acquisition Unit 141 The following describes the function of the acquisition unit 141 according to the sixth embodiment. The acquisition unit 141 acquires, as behavior-induced information, exercise information that indicates changes in the user's movements, measured from the user while he or she is exercising.

[0182] (4-2) Information Processing Unit 142 The function of the information processing unit 142 according to the sixth embodiment will be described. The information processing unit 142 generates, as output information, audio that indicates changes in the user's movements based on the acquired motion information. Specifically, the information processing unit 142 converts the motion information into corresponding audio and generates output information.

[0183] (4-3) Output Processing Unit 143 The following describes the function of the output processing unit 143 according to the sixth embodiment. The output processing unit 143 simultaneously outputs the output information generated by the information processing unit 142 and the model information stored in the storage unit 130 from the audio output device.

[0184] (5) Output Unit 150 The following describes the function of the output unit 150 according to the sixth embodiment. The output unit 150 outputs various types of information.

[0185] <6-3. Processing Flow> The functional configuration of the smart musical instrument 10 according to the sixth embodiment has been described above. Next, the processing flow according to the sixth embodiment will be described with reference to Fig. 13. Fig. 13 is a sequence diagram showing an example of the flow of practice support processing according to the sixth embodiment.

[0186] 13 , the user practices a sport (step S6101). The sensor device 6020 measures the user during the practice and acquires exercise information (step S6102). After acquisition, the sensor device 6020 transmits the acquired exercise information to the smart musical instrument 10 (step S6103).

[0187] The information processing unit 142 of the smart musical instrument 10 generates, as output information, audio that indicates changes in the user's movements during practice based on the motion information received from the sensor device 6020 (step S6104). The information processing unit 142 also acquires model information, which is audio of a model movement that the user is practicing, from the storage unit 130 (step S6105). Next, the output processing unit 143 transmits the output information and the model information to the audio output device (step S6106).

[0188] The audio output device 6030 simultaneously outputs sounds corresponding to the output information and the model information received from the smart musical instrument 10 (step S6107). The user simultaneously compares the sounds corresponding to the user's own movements and the model movements output from the audio output device 6030, and checks the difference between the user's movements and the model movements (step S6108).

[0189] As described above, the smart musical instrument 10 according to the sixth embodiment includes an acquisition unit 141 that acquires at least one of performance information indicating a performance by the user and behavior-induced information that is information different from the performance information and indicates information attributable to the user's behavior, an information processing unit 142 that generates output information indicating non-performance information corresponding to the user's psychology (request) and information related to music based on the acquired information of the performance information and the behavior-induced information, and an output processing unit 143 that outputs the output information. The acquisition unit 141 acquires, as behavior-induced information, motion information that indicates changes in the user's motion measured from the user while exercising, and the information processing unit 142 generates, as output information, sounds that indicate changes in the user's motion based on the acquired motion information.

[0190] With this configuration, even when the user is not playing a musical instrument in daily life, the user can input information resulting from the activity of exercising to the smart musical instrument 10, thereby obtaining information from the smart musical instrument 10 that reflects the user's desire to improve. This allows the user to receive support from the smart musical instrument 10 for practicing things other than the instrument, and allows the user to feel that the smart musical instrument 10 is there for them. Therefore, the smart musical instrument 10 according to the sixth embodiment can enhance the sense of symbiosis with music.

[0191] <<7. Modifications>> The embodiments have been described above. Next, modifications of the above-described embodiments will be described. Note that each modification described below may be applied to each embodiment alone, or may be applied in combination with each other. Furthermore, each modification may be applied in place of the configuration described in each embodiment, or may be applied in addition to the configuration described in each embodiment.

[0192] In the above-described embodiments, in the embodiments using trained models, examples have been described in which each trained model is stored in a server, but this is not limiting. Each trained model may be stored in the storage unit 130 of the smart musical instrument 10. In this case, the smart musical instrument 10 can obtain output from the trained model without communicating with the server.

[0193] Furthermore, in each of the above-described embodiments, the smart musical instrument 10 may have some or all of the functions of a terminal, device, server, etc. other than the smart musical instrument 10 (an example of an information processing device). For example, the smart musical instrument 10 according to the first embodiment may have the music information generation function of the music generation server 1030 instead. In this case, the music included in the music information is music automatically generated by the smart musical instrument 10. Furthermore, some or all of the functions of the smart musical instrument 10 may be possessed by a terminal, device, server, etc. other than the smart musical instrument 10. Furthermore, some or all of the functions of the smart musical instrument 10 and each server may be possessed by an external device (e.g., a smartphone or tablet terminal) located between the smart musical instrument 10 and each server.

[0194] Each embodiment has been described above. Note that a part or all of the smart musical instrument 10 (information processing device) in each of the above-described embodiments may be implemented by a computer. In this case, a program for implementing the functions may be recorded on a computer-readable recording medium, and the program may be loaded into a computer system and executed. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into a computer system. Furthermore, the term "computer-readable recording medium" may also include media that dynamically store programs for a short period of time, such as communication lines used when transmitting programs via networks such as the Internet or telephone lines, or media that store programs for a fixed period of time, such as volatile memory within a computer system that serves as a server or client. The program may be designed to implement a part of the functions described above, or may be capable of implementing the functions in combination with a program already stored in the computer system, or may be implemented using a programmable logic device such as an FPGA (Field Programmable Gate Array).

[0195] The embodiments of the present invention have been described in detail above with reference to the drawings, but the specific configuration is not limited to that described above, and various design changes can be made within the scope of the gist of the present invention.

[0196] 10...Smart musical instrument, 110...Communication unit, 120...Input unit, 130...Storage unit, 140...Control unit, 141...Acquisition unit, 142...Information processing unit, 143...Output processing unit, 150...Output unit, 1020...Sensor device, 1030...Music generation server, 1040...Reason estimation server, 1310...Communication unit, 1320...Storage unit, 1321...Music generation model, 1330...Control unit, 1331...Generation training unit, 1332...Music generation unit, 1410...Communication unit, 1420...Storage unit, 1421...Reason estimation model, 1430...Control unit, 1431...Reason training unit, 1432...Reason estimation unit, 2020...Imaging device, 2030...Music generation server, 2310...Communication unit communication unit, 2320...storage unit, 2321...music generation model, 2330...control unit, 2331...generation training unit, 2332...music generation unit, 3020...communication server, 3210...communication unit, 3220...storage unit, 3221...communication model, 3230...control unit, 3231...communication training unit, 3232...communication generation unit, 4020...user terminal, 4030...avatar management server, 4310...communication unit, 4320...storage unit, 4321...avatar management model, 4330...control unit, 4331...management training unit, 4332...avatar management unit, 6020...sensor device, 6030...audio output device

Claims

1. An information processing device comprising: an acquisition unit that acquires at least one of performance information indicating a performance by a user and behavior-induced information that is information different from the performance information and indicates information attributable to the user's behavior; an information processing unit that generates output information indicating non-performance information related to music in response to a request from the user based on the acquired information of the performance information and the behavior-induced information; and an output processing unit that outputs the output information.

2. The information processing device of claim 1, wherein the acquisition unit acquires an image selected by the user as the behavior-inducing information and further acquires the performance information, and the information processing unit generates a piece of music as the output information in accordance with the contents indicated by the acquired image and the performance information.

3. The information processing device according to claim 2, wherein the information processing unit generates a first piece of music in accordance with the content indicated by the image, and generates a second piece of music by correcting the first piece of music in accordance with the content indicated by the performance information as the output information.

4. The information processing device of claim 1, wherein the acquisition unit acquires information indicating the user's communication with the instrument as the performance information in the case of communication by playing the instrument, and acquires the information as the behavior-induced information in the case of communication other than by playing the instrument, and the information processing unit generates information indicating the reaction of the instrument to the communication from the user as the output information based on the acquired performance information or behavior-induced information.

5. The information processing device according to claim 4, wherein the output processing unit outputs the output information via an avatar that is a personification of a musical instrument.

6. The information processing device according to claim 5, wherein when the information indicating the reaction of the musical instrument generated by the information processing unit is a performance of the musical instrument, the output processing unit outputs the sound of the musical instrument being played by the avatar as the output information.

7. The information processing device according to claim 1, wherein the information processing section generates information indicating a reason for generating the output information based on a reason for generating the output information, by adding the information to the output information.

8. The information processing device of claim 7, wherein the acquisition unit acquires the output information selected by the user as the behavior-driven information, and the information processing unit generates, as the output information, information indicating a reason for generating the output information, which is the acquired behavior-driven information, based on a relationship between the output information and a reason for generating the output information.

9. The information processing device according to claim 8, wherein the output information is an automatically generated piece of music.

10. The information processing device of claim 1, wherein the acquisition unit acquires as the behavior-triggered information position information of an avatar that is a personification of a musical instrument, the position information changing as the user moves the musical instrument, and the information processing unit grows the avatar according to the location indicated by the acquired position information, and generates information indicating the degree of growth as the output information.

11. The information processing device of claim 1, wherein the acquisition unit acquires performance information indicating the user's performance using an instrument, and the information processing unit grows an avatar that anthropomorphizes the instrument based on the acquired performance information, and generates information indicating the degree of growth as the output information.

12. The information processing device according to claim 10 or 11, wherein the output processing unit outputs the musical instrument sound generated according to the degree of growth of the avatar.

13. An information processing device as described in claim 1, further comprising a memory unit that stores the performance information and the behavior-inducing information acquired by the acquisition unit as history information since the user started using the musical instrument, wherein the acquisition unit acquires trigger information that serves as a trigger for outputting the output information, and the information processing unit generates the output information from the history information stored in the memory unit based on the acquired trigger information.

14. The information processing device according to claim 13, wherein the information processing unit acquires the performance information of past users from the history information stored in the memory unit based on the trigger information, and the output processing unit outputs the acquired performance information.

15. The information processing device of claim 1, wherein the acquisition unit acquires motion information measured from the user while exercising and indicating a change in the user's motion as the behavior-induced information, and the information processing unit generates, as the output information, a sound indicating the change in the user's motion based on the acquired motion information.

16. An information processing device as described in claim 15, further comprising a memory unit that pre-stores audio indicating a change in a movement that serves as a model for the movement performed by the user as model information, wherein the output processing unit simultaneously outputs the output information generated by the information processing unit and the model information stored in the memory unit.

17. The information processing device according to claim 1, wherein the information processing unit generates the output information corresponding to the input information acquired by the acquisition unit using a trained model that has been trained to generate the output information from the relationship between the input information and the output information.

18. The information processing device according to claim 1, wherein the request is a demand according to the user's psychology.

19. An information processing method implemented by a computer, which comprises: acquiring at least one of performance information indicating a performance by a user and behavior-induced information which is information different from the performance information and indicates information attributable to the user's behavior; generating output information indicating non-performance information related to music in response to a request from the user based on the acquired information of the performance information and the behavior-induced information; and outputting the output information.

20. A program that causes a computer to acquire at least one of performance information indicating a performance by a user and behavior-induced information that is information different from the performance information and indicates information attributable to the user's behavior, generate output information indicating non-performance information related to music in response to a request from the user based on the acquired information of the performance information and the behavior-induced information, and output the output information.

Images