Performance sound generation method, performance sound generation device, and program

Abstract

This performance sound generation method comprises acquiring image information of a first musical instrument and sound information changed by an environmental change of the first musical instrument, acquiring performance operation information of a user, rendering an image of the first musical instrument on the basis of the image information, and generating a performance sound of the first musical instrument on the basis of the performance operation information and the sound information.

Description

Performance sound generation method, performance sound generation device, and program

[0001] An embodiment of the present invention relates to a performance sound generating method, a performance sound generating device, and a program.

[0002] The musical instrument 100 of Patent Document 1 has an operation unit 110 that accepts performance operations, a generation unit 120 that generates performance information representing the accepted performance operations, a detection unit 130 that detects a mobile terminal, and a control unit 170 that, when the mobile terminal is detected, starts a process of recording at least one of the performance information generated by the generation unit and a video image of the scene in which the performance operation is being performed onto at least one of the musical instrument and the mobile terminal.

[0003] The piano in Patent Document 2 generates performance information indicating the received performance operation, and if the performance operation does not involve pedal depression and the generated musical note is a single note, detects the sound production state of the single note (e.g., musical note waveform data). In this case, the piano detects the sound production state of the single note a predetermined time after the end of the performance operation to produce the single note, so as not to be affected by the musical note immediately preceding the single note. The piano transmits and outputs log data including sound production detection data indicating the detected sound production state and performance information to a server device. The server device analyzes the log data collected from the piano and generates and outputs notification information to the piano to notify the piano of the piano's status (e.g., whether tuning is required or when tuning is recommended).

[0004] The piano 10 of Patent Document 3 detects key information using a key sensor 151, pedal information using a pedal sensor 152, hammer information using a hammer sensor 153, plunger speed information using a plunger speed sensor 154, position information of the piano 10 using a GPS sensor 155, tilt information of the piano 10 using a tilt sensor 156, ambient temperature information using an ambient temperature sensor 157, ambient humidity information using an ambient humidity sensor 158, and plunger temperature information using a plunger temperature sensor 159, and transmits log data including these detected data to a server device. The server device analyzes the log data and generates notification information to notify the status of the piano 10 (for example, notification of whether tuning is necessary or when tuning is recommended), and transmits this information to the piano 10.

[0005] JP 2014-228750 JP 2014-228751 JP 2014-228752

[0006] One aspect of the present disclosure aims to provide a performance sound generation method that allows a user to perceive as if they are playing a musical instrument in any environment.

[0007] A performance sound generation method according to one embodiment of the present invention acquires image information of a first instrument and acoustic information that changes with changes in the environment of the first instrument, acquires performance operation information from a user, renders an image of the first instrument based on the image information, and generates a performance sound of the first instrument based on the performance operation information and the acoustic information.

[0008] According to one embodiment of the present invention, a user can perceive as if they are playing a musical instrument in any environment.

[0009] FIG. 1 is a configuration diagram of a performance sound generation system. FIG. 2 is a block diagram showing the configuration of a PC 1. FIG. 3 is a flowchart showing operations related to a performance sound generation method of the present embodiment. FIG. 4 is a configuration diagram of a performance sound generation system related to Modification 2. FIG. 5 is a configuration diagram of a performance sound generation system related to Modification 3. FIG. 6 is a configuration diagram of a performance sound generation system related to Modification 5. FIG. 7 is a configuration diagram of a performance sound generation system related to Modification 6.

[0010] 1 is a configuration diagram of a performance sound generation system according to this embodiment. The performance sound generation system of this embodiment includes a server 100 installed at a first location 10 and a PC installed at a second location 20. The first location 10 is, for example, a musical instrument store. The first location 10 includes a first musical instrument 2 for sale. The second location 20 is the home of a first performer 3, who is the user.

[0011] A first performer 3 at a second location 20 connects a second musical instrument 4 to a PC 1. In this embodiment, as an example, the first musical instrument 2 and the second musical instrument 4 are electric guitars. Note that in this embodiment, "performing" is not limited to playing an instrument, but also includes singing using a microphone.

[0012] 2 is a block diagram showing the configuration of the PC 1. In this embodiment, the PC 1 is an example of a performance sound generating device.

[0013] The PC 1 includes a display 31 , a user I / F 32 , a flash memory 33 , a processor 34 , a RAM 35 , a communication I / F 36 , a speaker (SP) 37 , and an audio I / F 38 .

[0014] The display 31 is composed of, for example, an LED, an LCD, or an OLED, and displays various information. The user I / F 32 is a touch panel that is laminated on the LCD or OLED of the display 31. Alternatively, the user I / F 32 may be a keyboard, a mouse, or the like. When the user I / F 32 is a touch panel, the user I / F 32, together with the display 31, constitutes a GUI (Graphical User Interface).

[0015] The communication I / F 36 includes a network interface and is connected to a network such as the Internet via a router (not shown). The communication I / F 36 is also connected to the camera 50.

[0016] The camera 50 captures video signals of the first performer 3 and the second musical instrument 4. The processor 34 performs signal processing on the video signals received from the camera 50.

[0017] The audio I / F 38 has an analog audio terminal. The audio I / F 38 is connected to a musical instrument or audio equipment such as a microphone via an audio cable and receives analog sound signals. In this embodiment, the audio I / F 38 of the PC 1 is connected to the second musical instrument 4 and receives analog sound signals related to performance sounds from the second musical instrument 4. The audio I / F 38 converts the received analog sound signals into digital sound signals. The audio I / F 38 also converts the digital sound signals into analog sound signals. The SP 37 plays sounds based on the analog sound signals.

[0018] The processor 34 is composed of a CPU, a DSP, a SoC (System-on-a-Chip), or the like, and reads a program stored in a flash memory 33, which is a storage medium, into a RAM 35 to control each component of the PC 1. The flash memory 33 stores the program of this embodiment.

[0019] The processor 34 performs signal processing on the digital sound signal received from the audio I / F 38. The processor 34 outputs the processed digital sound signal to the audio I / F 38. The audio I / F 38 converts the processed digital sound signal into an analog sound signal.

[0020] The SP 37 reproduces the analog sound signal output from the audio I / F 38 and reproduces the sound of the second musical instrument 4 .

[0021] The processor 34 of the PC 1 executes the performance sound generating method of this embodiment. Fig. 3 is a flowchart showing the operation of the performance sound generating method of this embodiment.

[0022] The processor 34 first acquires image information and sound information of the first musical instrument (S11). Specifically, the processor 34 receives image information 90 and sound information 91 of the first musical instrument 2 from the server 100.

[0023] The image information 90 is 3D model data of the first musical instrument 2. The model data of the first musical instrument 2 includes, for example, multiple polygon data and bone data for configuring the body, neck, strings, etc. of the first musical instrument 2. The multiple bone data that configure the model data of the first musical instrument 2 may have a link structure connected by multiple joint data. In particular, for musical instruments with a range of motion, it is preferable that the model data include a link structure. Furthermore, the image information 90 is not limited to 3D model data. The image information 90 may also be 2D image data. Furthermore, the image information 90 is not limited to still images and may also be video.

[0024] Such image information 90 is acquired by a camera 70, which is an example of a sensor connected to the server 100. 3D model data for the first musical instrument 2 is created in advance, but the camera 70 acquires a current external image of the first musical instrument 2. The server 100 adjusts the previously created 3D model data based on the current external image of the first musical instrument 2 acquired by the camera 70. For example, the server 100 may reflect changes in color due to aging, changes in the surface reflectivity of metal parts, etc. Alternatively, the server 100 may reflect differences in appearance between day and night within a single day. The server 100 recognizes the first musical instrument 2 from the external image of the first musical instrument 2 acquired by the camera 70 and determines identification information for the first musical instrument 2, such as its type and product name. The server 100 prepares a database in advance that associates external images of a large number of musical instruments with their identification information, and acquires the corresponding identification information for the first musical instrument 2 from the external image of the first musical instrument 2 captured by the camera 70. Alternatively, the server 100 may prepare a trained model, for example, trained using a DNN or the like, to determine the relationship between the appearance image of the first musical instrument 2 and its identification information, and input the appearance image into the trained model to determine the identification information. In this case, the server 100 acquires, as a training phase, a large number of data sets of, for example, musical instrument appearance images and identification information. Based on the acquired appearance images and identification information, the server 100 trains a predetermined model to determine the relationship between the appearance image and the identification information. As an execution phase, the server 100 inputs the appearance image of the first musical instrument 2 received from the camera 70 into the trained model to obtain the identification information. This allows the server 100 to determine the image information 90 and sound information 91 of the first musical instrument 2 and transmit them to the PC 1A in S11.

[0025] The acoustic information 91 includes data that models the sound of the first musical instrument 2 as a digital sound source. The sound of the first musical instrument 2 changes in response to environmental changes. For example, the properties of wood, which is the main material used in the body of a guitar, change over time. The magnets used in the pickups also change over time. The sound of the first musical instrument 2 changes over time. The sound of the first musical instrument 2 also changes depending on the temperature, humidity, etc., of the storage environment. The acoustic information 91 not only models the sound of a new product, but also includes data that models the sound that has changed in response to these environmental changes, and changes over time.

[0026] In the process of S11, the processor 34 may receive from the first performer 3 the time point (for example, the present, one year ago, three years ago, etc.) from which the image information 90 or audio information 91 is to be acquired. The processor 34 acquires from the server 100 the image information 90 or audio information 91 corresponding to the received time point.

[0027] The acoustic information 91 may also include information about the playback environment. This information about the playback environment includes, for example, information about the acoustic equipment (effectors, amplifiers, speakers, etc.) connected to the first musical instrument 2 and information about the reverberation of the playback space. The sound of the first musical instrument 2 also changes depending on the connected acoustic equipment and the reverberation of the playback space. For example, the sound of the first musical instrument 2 differs depending on the environment, such as a studio playing room, a concert hall, outdoors, etc. The acoustic information 91 may include information about these various playback environments.

[0028] Next, the processor 34 acquires performance operation information of the first performer 3 (S12). For example, in the case of a guitar, performance operation information includes information indicating which fret is being pressed, the timing at which the fret is pressed, the timing at which the fret is released, information indicating which string is being picked, the picking timing, the picking speed, whether or not a mute operation is performed, etc. Furthermore, in the case of a keyboard instrument such as a synthesizer, performance operation information includes time parameters such as pitch (note number), tone color, attack, decay, sustain, and release.

[0029] In this embodiment, the performance operation information is acquired by the performance operation of the first performer 3 on the second instrument 4.

[0030] The processor 34 acquires the performance operation information based on the video signal from the camera 50. Alternatively, the processor 34 may acquire the performance operation information by acquiring the motion data of the performer using, for example, a motion sensor.

[0031] The processor 34 can also acquire instrument operation information from sensors mounted on the instrument. For example, in the case of a guitar, the sensors mounted on the instrument are fret sensors attached to each fret. The processor 34 acquires sensor signals for each fret to acquire instrument operation information. Alternatively, in the case of an electronic instrument in which the strings to be picked are replaced with sensors, the processor 34 acquires sensor signals to acquire instrument operation information.

[0032] The processor 34 may also extract features of the digital sound signal (sound signal of the second instrument 4) received from the audio I / F 38 and compare them with features corresponding to previously detected operation information to obtain performance operation information. Alternatively, the processor 34 may prepare a trained model, for example, trained using a DNN or the like, to determine the relationship between the sound signal and the performance operation information, and input the sound signal into the trained model to obtain the performance operation information. In the training phase, the processor 34 may acquire a data set of the sound signal and the performance operation information from a server or the like. Alternatively, in the training phase, the processor 34 may acquire sensor signals for each fret to obtain performance operation information, and then acquire the sound signal received at that timing. Based on the acquired sound signal and performance operation information, the processor 34 trains a predetermined model to determine the relationship between the sound signal and the performance operation information. In the execution phase, the processor 34 inputs the sound signal received from the second instrument 4 into the trained model to obtain the performance operation information.

[0033] The processor 34 then renders an image of the first musical instrument 2 based on the image information 90 (S13). More specifically, the processor 34 controls the 3D model data of the first musical instrument 2 included in the image information 90 using the performance operation information. The processor 34 may also control 3D model data of a certain performer 80. The model data of the performer 80 includes, for example, multiple polygon data and bone data for constituting the performer's face, torso, arms, fingers, legs, etc. The multiple bone data have a link structure connected by multiple joint data. The position information of each bone data in the model data is defined by the motion data. The processor 34 controls the position information of the model data of the performer 80 based on the performance operation information of the performer. The processor 34 renders the 3D model data of the performer 80 and controls the position information of the 3D model data based on the performance operation information. The processor 34 displays an image related to the rendered 3D model data (an image of the first musical instrument 2 and an image of the performer 80 included in the image information 90) on the display 31.

[0034] The processor 34 then generates a performance sound of the first musical instrument 2 based on the performance operation information and the acoustic information 91 (S14). The performance operation information corresponds to parameters for synthesizing the sound of the sound source of the acoustic information 91. The processor 34 synthesizes the sound of the sound source (the guitar sound source of the first musical instrument 2 in this embodiment) based on the performance operation information of the first performer 3.

[0035] The processor 34 may perform signal processing based on information about the playback environment included in the acoustic information 91. For example, if the acoustic information 91 includes information about reverberation, the processor 34 may perform signal processing on the sound signal of the synthesized sound by convolving impulse response data of the playback environment as a process for reproducing the reverberation of the playback environment. The processor 34 may also perform filter processing on the sound signal of the synthesized sound to simulate the acoustic devices (effectors, amplifiers, speakers, etc.) connected to the first musical instrument 2. Specifically, the information about the playback environment includes parameters of a digital signal processing block that simulates, as a digital filter, the input-to-output characteristics of each acoustic device connected to the first musical instrument 2. The processor 34 performs signal processing on the sound signal of the synthesized sound using the parameters indicated in the information about the acoustic devices. This allows the processor 34 to reproduce, for the synthesized sound, the input-output characteristics of the acoustic devices (effectors, amplifiers, speakers, etc.) connected to the first musical instrument 2.

[0036] The processor 34 outputs the generated sound signal related to the performance sound of the first instrument 2 to the speaker 37 via the audio I / F 38. As a result, the performance sound of the first instrument 2 is reproduced from the speaker 37 in response to the performance operation of the first performer 3 on the second instrument 4. Note that the server 100 may execute the operations shown in S13 and S14 in Fig. 3 and transmit the video and audio signals generated in S13 and S14 to the PC 1. In this case, the PC 1 receives the video and audio signals from the server 100 via the network and reproduces the received performance sound of the first instrument 2 in response to the performance operation of the user on the second instrument 4.

[0037] In this way, the first performer 3 can use his / her own second instrument 4 at home to try out the first instrument 2 in the store and listen to the performance sound of the first instrument 2 from the comfort of his / her own home. Users of the performance sound generation method of this embodiment can have the customer experience of perceiving as if they were playing a favorite instrument other than the second instrument 4 that they are actually playing. More specifically, users of the performance sound generation method of this embodiment can have the customer experience of being able to play a favorite instrument even remotely.

[0038] (Modification 1) The performance sound generation system of modification 1 records non-fungible tokens (hereinafter referred to as NFTs) corresponding to image information 90 and sound information 91 in a digital ledger on the Internet.

[0039] For example, a musical instrument store, which is a first location 10 in FIG. 1 , records NFTs corresponding to image information 90 and sound information 91 of a first musical instrument 2, which is a vintage item, in a digital ledger. This allows the musical instrument store to authenticate and associate information about the appearance and sound of the musical instrument it is selling with the NFT. The musical instrument store can, for example, sell the image information 90 and sound information 91 of the first musical instrument 2, which is a vintage item, authenticated with the NFT, and receive payment in return. For example, the musical instrument store may offer a free trial period during which it provides image information 90 and sound information 91 that are not NFT-compatible, and then provide the image information 90 and sound information 91 authenticated with the NFT after receiving a fee.

[0040] (Modification 2) Fig. 4 is a diagram showing the configuration of a performance sound generating system according to Modification 2. Components common to those in Fig. 1 are given the same reference numerals, and a description thereof will be omitted.

[0041] In the performance sound production system according to the second modification, a PC 1 at a second location 20 and a PC 1A installed at a third location 30 are connected via a network. The configuration of the PC 1A is the same as that of the PC 1 shown in FIG. 2.

[0042] A microphone 8, a camera 50, and a camera 70 are connected to PC1A. At third location 30, second performer 7 sings using microphone 8. PC1A transmits an audio signal related to the singing sound received by microphone 8 to PC1. PC1A also transmits a video signal of second performer 7 received from camera 50 to PC1.

[0043] PC1 plays back the sound signal related to the singing sound of second performer 7 received from PC1A. PC1 displays the video related to the 3D model data rendered in S13 of Fig. 3 (the video of first musical instrument 2 and the video of performer 80 included in image information 90) and the video of second performer 7 received from PC1A.

[0044] PC 1 transmits the performance operation information generated in S12 of FIG. 3 to PC 1A. PC 1A executes the operations shown in S13 and S14 of FIG. 3 based on the received performance operation information. PC 1A displays the video associated with the 3D model data rendered in S13 (the video of the first musical instrument 2 and the video of the performer 80 included in the image information 90) and the video of the second performer 7 received from camera 50. PC 1A also plays back the sound signal associated with the performance sound of the first musical instrument 2 generated in S14. Alternatively, PC 1 may execute the operations shown in S13 and S14 of FIG. 3 and transmit the video and the sound signal associated with the performance sound of the first musical instrument 2 generated in S13 and S14 to PC 1A via the network. PC 1A plays back the received video and sound signals.

[0045] In this way, the performance sound generation system according to the second modification allows the first performer 3 at the second location 20 and the second performer 7 at the third location 30 to play a remote ensemble. In the performance sound generation system according to the second modification, the first performer 3 can play the first instrument 2 at the third location 30, which is a studio, while at home using his / her own second instrument 4, and play an ensemble with the second performer 7.

[0046] (Modification 3) Fig. 5 is a diagram showing the configuration of a performance sound generating system according to Modification 3. Components common to those in Fig. 4 are given the same reference numerals, and a description thereof will be omitted.

[0047] In the performance sound generation system according to the third modification, the first instrument 2 is an electric guitar owned by the first performer 3, and the second instrument 4 is an electric guitar installed at the third location 30, which is a studio.

[0048] A camera 70 is connected to PC 1. A microphone 8 and two cameras 50 are connected to PC 1A.

[0049] 3. PC 1A displays the video associated with the 3D model data rendered in S13 (the video of the first musical instrument 2 and the video of the performer 80 included in the image information 90) and the video of the second performer 7 received from camera 50. PC 1A also plays back the sound signal associated with the performance sound of the first musical instrument 2 generated in S14.

[0050] In the performance sound generation system according to the third modification, the first performer 3 can play the first instrument 2 at the second location 20 in his / her home while using the second instrument 4 at the third location 30, which is a studio, to perform in an ensemble with the second performer 7. Therefore, the first performer 3 can perform using the sound of his / her own electric guitar wherever he / she is located, without having to carry his / her own electric guitar with him / her.

[0051] (Variation 4) For musical instruments that are composed of multiple elements, image information 90 and sound information 91 may be provided for each element. For example, a saxophone has elements such as a body, neck, mouthpiece, ligature, and reed. Image information 90 and sound information 91 are provided for each of these elements such as the body, neck, mouthpiece, ligature, and reed.

[0052] 3, the PC 1 renders an image of the saxophone based on a combination of a plurality of pieces of image information 90. Also, in the operation of S14 in FIG. 3, the PC 1 generates a performance sound based on a combination of a plurality of pieces of sound information 91.

[0053] This allows the user to change the appearance and tone of the instrument by combining multiple elements that make up the instrument.

[0054] (Modification 5) Fig. 6 is a diagram showing the configuration of a performance sound generating system according to Modification 5. Components common to those in Fig. 1 are given the same reference numerals, and a description thereof will be omitted.

[0055] In the musical performance sound production system according to the fifth modification, a first musical instrument 2 is installed at a second location 20. A camera 50 and a camera 70 are connected to a PC 1.

[0056] The PC 1 executes the operations S11 to S14 shown in Fig. 3. The image information 90 and the sound information 91 may be stored in the flash memory 33 of the PC 1, or may be stored in another device (for example, the server 100) and downloaded by the PC 1 each time.

[0057] In this case, too, the user can have the customer experience of perceiving as if he or she is playing a favorite instrument (for example, the first instrument 2) other than the second instrument 4 that he or she is actually touching.

[0058] (Modification 6) Fig. 7 is a diagram showing the configuration of a performance sound generating system according to Modification 6. Components common to those in Fig. 6 are given the same reference numerals, and a description thereof will be omitted.

[0059] In the performance sound generation system according to the sixth modification, a first musical instrument 2 is installed at a second location 20. A first performer 3 plays the first musical instrument 2. The first musical instrument 2 and a camera 50 are connected to a PC 1.

[0060] The PC 1 executes the operations S11 to S14 shown in Fig. 3. The image information 90 and the sound information 91 may be stored in the flash memory 33 of the PC 1, or may be stored in another device (for example, the server 100) and downloaded by the PC 1 each time.

[0061] In the sixth variation, the first performer 3 plays the first musical instrument 2, and the PC 1 renders an image of the first musical instrument 2 and 3D model data of the performer 80. The PC 1 also generates the performance sound of the first musical instrument 2 based on the sound information 91, in which the first performer 3 plays the first musical instrument 2.

[0062] This allows users to have a new customer experience, perceiving that they are playing a musical instrument in any environment. For example, if the PC 1 acquires acoustic information 91 that models the sound of the first musical instrument 2 when it was new, it can generate the performance sound of the first musical instrument 2 when it was new. Therefore, the user can play the performance sound of the first musical instrument 2 when it was new. Conversely, for example, if the PC 1 acquires acoustic information 91 that models the past sound of the first musical instrument 2, it can generate the past performance sound (vintage sound) even with a new first musical instrument 2.

[0063] Furthermore, if the acoustic information 91 includes information about reverberation, the PC 1 may display an image of a virtual concert hall or the like on the display 31 and perform processing to reproduce the reverberation of the playback environment of the concert hall, etc. This allows the user to have a new customer experience, for example, by perceiving a live performance in a longed-for live music venue or concert hall that no longer exists.

[0064] The description of the present embodiment should be considered to be illustrative in all respects and not restrictive. The scope of the present invention is defined by the claims, not by the above-described embodiments. Furthermore, the scope of the present invention includes the scope equivalent to the claims.

[0065] For example, the present invention may be a performance sound generation method that acquires image information of a first instrument and audio information of the first instrument, acquires performance operation information of a user on a second instrument, renders an image of the first instrument based on the image information, and generates a performance sound of the first instrument based on the performance operation information and the audio information.

[0066] In this case, too, by playing the second instrument at home or elsewhere, the sound of the first instrument installed in a remote location such as a music store can be reproduced, allowing users to play their favorite instrument remotely.

[0067] As shown in Figure 6 or 7, the present invention does not require a network configuration, and the image information 90 and the acoustic information 91 do not need to be acquired via a network.

[0068] 2: First instrument, 3: First performer, 4: Second instrument, 7: Second performer, 8: Microphone, 10: First location, 20: Second location, 30: Third location, 31: Display, 32: User I / F, 33: Flash memory, 34: Processor, 35: RAM, 36: Communication I / F, 37: Speaker, 38: Audio I / F, 50: Camera, 70: Camera, 80: Performer, 90: Image information, 91: Sound information, 100: Server

Claims

1. A performance sound generation method comprising: acquiring image information of a first instrument and audio information of the first instrument that changes with changes in the environment; acquiring performance operation information from a user; rendering an image of the first instrument based on the image information; and generating a performance sound of the first instrument based on the performance operation information and the audio information.

2. The performance sound generating method according to claim 1, wherein the performance operation information is acquired by a performance operation of the user on a second musical instrument.

3. The performance sound generating method according to claim 2, wherein the image information or the audio information changes over time.

4. A method for generating a performance sound as described in claim 2 or claim 3, further comprising the steps of: transmitting the performance sound of the first instrument to an information processing device used by the user via a network; receiving the performance sound of the first instrument via the network by the information processing device; and reproducing the performance sound of the first instrument in response to the performance operation of the user on the second instrument.

5. A performance sound generating method as described in any one of claims 1 to 3, further comprising acquiring information regarding a playback environment, and performing signal processing on the generated performance sound of the first instrument based on the information regarding the playback environment.

6. The performance sound generating method according to any one of claims 1 to 3, wherein the image information and the acoustic information of the first musical instrument are acquired via a network.

7. The performance sound generating method according to any one of claims 1 to 3, further comprising the step of: reproducing the performance sound of the first musical instrument.

8. A method for generating a musical performance sound according to any one of claims 1 to 3, further comprising the step of recording non-fungible tokens corresponding to the image information and the audio information.

9. The method for generating a performance sound according to claim 8, further comprising the steps of: authenticating the image information and the sound information of the first musical instrument with the non-fungible token; and providing the authenticated image information and the sound information of the first musical instrument.

10. A performance sound generating device having a processor that performs the following processes: acquire image information of a first musical instrument and audio information of the first musical instrument that changes with changes in the environment; acquire performance operation information by a user; render an image of the first musical instrument based on the image information; and generate a performance sound of the first musical instrument based on the performance operation information and the audio information.

11. The performance sound generating device according to claim 10, wherein the performance operation information is acquired by a performance operation of the user on a second musical instrument.

12. The performance sound generating device according to claim 11, wherein the acoustic information changes over time.

13. The performance sound generating device according to any one of claims 10 to 12, wherein the image information includes 3D model data.

14. The performance sound generating device according to any one of claims 10 to 12, wherein the image information is acquired by a sensor.

15. The performance sound generating device according to any one of claims 10 to 12, wherein the performance operation information is acquired by a sensor.

16. The performance sound generating device according to any one of claims 10 to 12, wherein the processor records non-fungible tokens corresponding to the image information and the sound information.

17. A program that causes an information processing device to execute the following processes: acquire image information of a first instrument and audio information of the first instrument that changes with changes in the environment; acquire performance operation information by a user; render an image of the first instrument based on the image information; and generate a performance sound of the first instrument based on the performance operation information and the audio information.

Images