DEVICE, ENSEMBLE SYSTEM, SOUND REPRODUCTION METHOD, AND PROGRAM - Patent application

Abstract

A device that is provided in a first venue when an ensemble is performing remotely at the first venue and a second venue. The device has an estimation unit that inputs, to a performance audio estimation model, performance audio picked up by the device in the second venue and estimates future performance audio in the performance audio.

Description

[Technical field]

[0001] The present invention relates to an apparatus, an ensemble system, a sound reproduction method, and a program. [Background technology]

[0002] There is a system that delivers live video footage of singing or musical performance (see, for example, Patent Document 1). In this system, performers such as singers and musicians perform in separate locations. Each performance location is equipped with a camera. The center synthesizes the footage captured by each camera and delivers it to receiving terminals as a distribution video. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2008-131379 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, when performers are far apart, they must receive and hear each other's sounds via communication lines. When using communication lines, delays in transmission can occur, and the sound of the other person may be heard with a delay. For this reason, when performers are far apart, it can be difficult for them to play together naturally.

[0005] The present invention has been made in view of the above circumstances, and an object of the present invention is to reproduce sound received via a communication line without delay. [Means for solving the problem]

[0006] One aspect of the present invention is an apparatus provided in a first venue when a remote ensemble is performed between a first venue and a second venue, the apparatus having an estimation unit that inputs performance sounds picked up by an apparatus provided in the second venue into a performance sound estimation model and estimates future estimated performance sounds from the performance sounds, the performance sound estimation model being a trained model that has been trained to estimate the estimated performance sounds from the input performance sounds by learning sound signals corresponding to the performance sounds.

[0007] According to one aspect of the present invention, there is provided an ensemble system for realizing a remote ensemble performance at a first venue and a second venue, the system comprising a first terminal device provided at the first venue and a second terminal device provided at the second venue, the first terminal device having a first acquisition unit for acquiring a first performance sound at the first venue, a first transmission unit for transmitting the first performance sound to the second terminal device, a first reception unit for receiving a second performance sound at the second venue from the second terminal device, a first estimation unit for estimating a future second performance estimated sound for the second performance sound by inputting the second performance sound received by the first reception unit to a second performance sound estimation model, and a first sound output unit for outputting the second performance estimated sound, the first performance sound estimation model is a trained model trained to estimate the first performance estimated sound from the input first performance sound by learning a first sound signal corresponding to the first performance sound, and the second performance sound estimation model is a trained model trained to estimate the second performance estimated sound from the input second performance sound by learning a second sound signal corresponding to the second performance sound.

[0008] Another aspect of the present invention is a sound reproduction method performed by a computer device installed in a first venue when a remote ensemble is performed between a first venue and a second venue, the sound reproduction method comprising: inputting performance sounds picked up by a device installed in the second venue into a performance sound estimation model to estimate future performance estimated sounds for the performance sounds; and the performance sound estimation model is a trained model that has been trained to estimate the estimated performance sounds from the input performance sounds by learning sound signals corresponding to the performance sounds.

[0009] Another aspect of the present invention is a program for causing a computer device installed in a first venue to input performance sounds picked up by a device installed in the second venue into a performance sound estimation model when a remote ensemble is performed between a first venue and a second venue, and to estimate a future estimated performance sound from the performance sounds, wherein the performance sound estimation model is a trained model that has been trained to estimate the estimated performance sound from the input performance sounds by learning sound signals corresponding to the performance sounds. Effect of the Invention

[0010] Sound received via a communication line can be played back without delay. [Brief description of the drawings]

[0011] [Figure 1] 1 is a schematic diagram showing an overview of an ensemble system 1 according to an embodiment. [Diagram 2] 1 is a block diagram showing an example of the configuration of an ensemble system 1 according to an embodiment. [Diagram 3] FIG. 2 is a diagram illustrating an example of a trained model 120 according to an embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of a trained model 120 according to an embodiment. [Diagram 5] FIG. 2 is a diagram illustrating an example of a trained model 120 according to an embodiment. [Figure 6] 2 is a sequence diagram illustrating the flow of processing performed by the ensemble system 1 according to the embodiment. FIG. [Figure 7]10 is a flowchart illustrating the flow of processing performed by the player terminal 10 according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] The ensemble system 1 according to the embodiment will be described below with reference to the drawings. In the following, an example will be described in which performers located far away from each other hold a session (remote ensemble) using the ensemble system 1. However, the present invention is not limited to this, and the ensemble system 1 according to the present embodiment can be applied to synthesizing any content other than sound.

[0013] 1 is a schematic diagram showing an outline of an ensemble system 1 according to an embodiment. The ensemble system 1 is a system that transmits performance sounds by a performer to other performers located remotely in real time.

[0014] As shown in FIG. 1, in ensemble system 1, sounds related to a performance in venue E1 (first performance sounds) are picked up by microphone MC1 and transmitted to venue E2, which is the session partner, via communication network NW.

[0015] In venue E2, the first performance sound received via the communication network NW is output from speaker SP2. In venue E2, the performance sound (second performance sound) in venue E2 is picked up by microphone MC2 and transmitted to venue E1 via the communication network NW. In venue E1, the second performance sound received via the communication network NW is output from speaker SP1. In ensemble system 1, the first performance sound and the second performance sound are transmitted to distribution server 20, mixed, and distributed to viewer terminal 30 via distribution server 20.

[0016] In the ensemble system 1, future performance sounds are estimated from the performance sounds of the session partner received via the communication network NW. Here, the future performance sounds are sounds that will be played at a performance position (T+Δt) in the future from the performance position T in the received performance sounds of the session partner.

[0017] Specifically, the second performance sound is received at the venue E1, and a future performance sound of the second performance sound is estimated based on the received second performance sound, while the first performance sound is received at the venue E2, and a future performance sound of the first performance sound is estimated based on the received first performance sound.

[0018] A trained model is used for the estimation. The trained model is a model that has trained sound signals related to a performance sound. The trained model is trained so as to estimate a future performance sound of the input performance sound from the input performance sound.

[0019] Specifically, the trained model is created by performing machine learning (e.g., deep learning) of the trained model using the sound signal of the played sound as training data. The trained model is, for example, a neural network or a multi-layer tree model.

[0020] The sound signal of the learning data is, for example, an audio signal obtained by collecting the sound of a musical instrument being played by a microphone. The sound signal includes time-series data in which instruction data indicating the content of the performance and time data indicating the time point at which the instruction data is generated are arranged. The instruction data specifies a pitch (note number) and intensity (velocity) to instruct various events such as sound generation and sound muting. The time data specifies, for example, the interval (delta time) between adjacent instruction data.

[0021] That is, in the ensemble system 1, a performance sound received via the communication network NW is input to the trained model. The trained model estimates and outputs a future performance sound from the input performance sound. The future performance sound estimated by the trained model is output from the speaker.

[0022] Specifically, the second performance sound is received at the venue E1, and the received second performance sound is input to a trained model (second performance sound estimation model). The second performance sound estimation model is a model that has trained a sound signal related to the second performance sound. The second performance sound estimation model estimates future performance sounds from the input second performance sound. The performance sound estimated by the second performance sound estimation model is output from the speaker SP1.

[0023] The first performance sound is received at the venue E2, and the received first performance sound is input to a trained model (first performance sound estimation model). The first performance sound estimation model is a model that has trained a sound signal related to the first performance sound. The first performance sound estimation model estimates future performance sounds from the input first performance sound. The performance sound estimated by the first performance sound estimation model is output from the speaker SP2.

[0024] As a result, the ensemble system 1 of this embodiment can estimate and output future performance sounds from performance sounds received via the communication network NW. Therefore, even if a performance sound is received at a performance position T that is delayed from the actual performance position (T+Δt) due to a transmission delay, it is possible to estimate and output the performance sound at the actual performance position (T+Δt). Therefore, it is possible to play back sounds received via a communication line without delay.

[0025] Here, the sound signal of the learning data used for learning may be determined arbitrarily. The sound signal of the learning data may be at least a sound signal corresponding to the performance sound to be estimated, but is preferably a sound played in a similar performance style to the performance sound to be estimated. This is because the accuracy of estimation can be improved by learning a performance sound with a similar performance style.

[0026] For example, the sound signal of the learning data is preferably a performance sound by a performer who actually plays in a real remote ensemble. Also, the sound signal of the learning data is preferably a performance sound using an instrument that is actually played in a real remote ensemble. The sound signal of the learning data is, for example, a performance sound played in a rehearsal (a rehearsal sound source). By using the rehearsal sound source, the performance sound in the real remote ensemble can be accurately estimated.

[0027] Fig. 2 is a block diagram showing an example of the configuration of ensemble system 1 according to an embodiment. Here, an example will be described in which three performer terminals 10-1 to 10-3 perform remotely. However, the present invention is not limited to this example. Ensemble system 1 can be applied to cases in which multiple performer terminals 10 (performer terminals 10-1 to 10-N, N is a natural number different from 1) perform remotely.

[0028] 1, ensemble system 1 includes, for example, three performer terminals 10-1 to 10-3, a distribution server 20, and a viewer terminal 30. Note that in ensemble system 1, a plurality of viewer terminals 30 may be provided.

[0029] The performer terminal 10-1 is a computer device such as a smartphone, a mobile terminal, a tablet, or a PC (Personal Computer) that is installed in the venue E1 in Fig. 1. The speaker unit 15 of the performer terminal 10-1 corresponds to the speaker SP1 in Fig. 1. The microphone unit 16 of the performer terminal 10-1 corresponds to the microphone MC1 in Fig. 1.

[0030] The performer terminal 10-2 is a computer device such as a smartphone, mobile terminal, tablet, or PC that is installed in venue E2 in Fig. 1. The speaker unit 15 equipped in the performer terminal 10-2 corresponds to the speaker SP2 in Fig. 1. The microphone unit 16 equipped in the performer terminal 10-2 corresponds to the microphone MC2 in Fig. 1. Although omitted in Fig. 1, the same is true for the performer terminal 10-3. In the following explanation, when there is no need to distinguish between the performer terminals 10-1 to 10-3, they will simply be referred to as "performer terminal 10."

[0031] In the ensemble system 1, the performer terminals 10, the distribution server 20, and the viewer terminals 30 are communicatively connected via a communication network NW. The communication network NW is, for example, a wide area network, i.e., a WAN (Wide Area Network), the Internet, or a combination of these.

[0032] The performer terminal 10 includes, for example, a communication unit 11, a storage unit 12, a control unit 13, a display unit 14, a speaker unit 15, and a microphone unit 16.

[0033] The communication unit 11 communicates with the distribution server 20. The storage unit 12 is configured by a storage medium such as a HDD, a flash memory, an EEPROM (Electrically Erasable Programmable Read Only Memory), a RAM (Random Access read / write Memory), or a ROM (Read Only Memory), or a combination of these. The storage unit 12 stores programs for executing various processes of the performer terminal 10, and temporary data used when performing various processes. The storage unit 12 stores, for example, a trained model 120. The trained model 120 is information necessary for constructing a trained model. The information necessary for constructing a trained model is the configuration of the trained model, the setting values ​​of parameters to be used, and the like. For example, when the trained model is configured as a Convolutional Neural Network (CNN) having an input layer, an intermediate layer, and an output layer, the configuration of the trained model is information indicating the number of units in each layer, the number of intermediate layers, an activation function, and the like. The parameters to be used are information indicating the coupling coefficients and weights that couple the nodes of each layer.

[0034] The trained model 120 will be described with reference to Figs. 3 to 5. Fig. 3 is a diagram showing an example of trained model 120-1 stored in performer terminal 10-1. Fig. 4 is a diagram showing an example of trained model 120-2 stored in performer terminal 10-2. Fig. 5 is a diagram showing an example of trained model 120-3 stored in performer terminal 10-3. In the following description, when there is no need to distinguish between trained models 120-1 to 120-3, they will simply be referred to as "trained model 120."

[0035] As shown in Figs. 3 to 5, the trained model 120 includes items such as a target venue number, a performance type, and a trained model. The target venue number is identification information such as a number that uniquely identifies the venue where the performance will be performed. The performance type is information indicating the type of performance to be performed at the venue identified by the target venue number, for example, the instrument to be played. The trained model is a trained model that corresponds to the performance sound of the performance to be performed at the venue identified by the target venue number.

[0036] In the example of FIG. 3, it is shown that a second learned model and a third learned model are stored in the learned model 120-1. The second learned model is a model that estimates a future performance sound corresponding to a trumpet performance sound performed at a venue identified by the target venue No. (2). The third learned model is a model that estimates a future performance sound corresponding to a trumpet performance sound performed at a venue identified by the target venue No. (3). The venue identified by the target venue No. (1) corresponds to the venue where the performer terminal 10-1 is installed. The venue identified by the target venue No. (2) or the target venue No. (3) corresponds to the venue where the session partner is located.

[0037] In the example of FIG. 4, it is shown that a first learned model and a third learned model are stored in the learned model 120-2. The first learned model is a model that estimates a future performance sound corresponding to a trumpet performance sound performed at a venue identified by a target venue No. (1). The third learned model is a model that estimates a future performance sound corresponding to a trumpet performance sound performed at a venue identified by a target venue No. (3). The venue identified by a target venue No. (2) corresponds to the venue where the performer terminal 10-2 is installed. The venue identified by a target venue No. (1) or a target venue No. (3) corresponds to the venue where the session partner is located.

[0038] In the example of FIG. 5, it is shown that a first learned model and a second learned model are stored in the learned model 120-3. The first learned model is a model that estimates a future performance sound corresponding to a trumpet performance sound performed at a venue identified by a target venue No. (1). The second learned model is a model that estimates a future performance sound corresponding to a trumpet performance sound performed at a venue identified by a target venue No. (2). The venue identified by a target venue No. (3) corresponds to the venue where the performer terminal 10-3 is installed. The venue identified by a target venue No. (1) or a target venue No. (2) corresponds to the venue where the session partner is located.

[0039] As shown in FIGS. 3 to 5, the trained model 120 stores a trained model for estimating a performance sound that will be a session partner.

[0040] Returning to the explanation of Fig. 1, the control unit 13 is realized by causing a CPU (Central Processing Unit) provided as hardware in the performer terminal 10 to execute a program. The control unit 13 comprehensively controls the performer terminal 10. The control unit 13 controls each of the communication unit 11, the storage unit 12, the display unit 14, the speaker unit 15, and the microphone unit 16.

[0041] The control unit 13 includes, for example, an acquisition unit 130, an estimation unit 131, an output unit 132, and a distribution unit 133. The acquisition unit 130 acquires a performance sound of a session partner. The acquisition unit 130 outputs the acquired performance sound to the estimation unit 131.

[0042] The estimation unit 131 estimates a future performance sound by inputting the performance sound acquired from the acquisition unit 130 to the trained model. The estimation unit 131 outputs the estimated performance sound to the output unit 132.

[0043] The output unit 132 outputs the performance sound acquired from the estimation unit 131 to the speaker unit 15. As a result, the future performance sound of the session partner is emitted from the speaker unit 15.

[0044] When there are multiple session partners, the output unit 132 may output a sound obtained by mixing the future performance sounds of the performance sounds of the respective session partners.

[0045] The distribution unit 133 transmits the performance sound picked up by the microphone unit 16 via the communication unit 11 to the performer terminal 10 of the session partner and the distribution server 20.

[0046] The display unit 14 includes a display device such as a liquid crystal display, and displays images such as video related to the performance of the session partner under the control of the control unit 13. The speaker unit 15 outputs the performance sound of the session partner under the control of the control unit 13.

[0047] The distribution server 20 is a computer device that distributes video and audio related to a performance. The distribution server 20 is, for example, a server device, a cloud, a PC, or the like.

[0048] The distribution server 20 includes, for example, a communication unit 21, a storage unit 22, and a control unit 23. The communication unit 21 communicates with each of the performer terminals 10 and the viewer terminal 30.

[0049] The storage unit 22 is configured by, for example, a storage medium such as a HDD, a flash memory, an EEPROM, a RAM, a ROM, or a combination of these. The storage unit 22 stores programs for executing various processes of the distribution server 20 and temporary data used when performing various processes.

[0050] The storage unit 22 stores, for example, distribution information 220. The distribution information 220 is information related to the sound to be distributed. The distribution information 220 is, for example, information indicating a list of the viewer terminals 30 to which the sound is to be distributed and the contents to be distributed.

[0051] The control unit 23 is realized by executing a program on a CPU provided as hardware in the distribution server 20. The control unit 23 includes, for example, an acquisition unit 230, a synthesis unit 231, and a distribution unit 232.

[0052] The acquiring unit 230 acquires performance sounds from each of the performer terminals 10. The acquiring unit 230 outputs information indicating each of the acquired performance sounds to the synthesis unit 231.

[0053] The synthesis unit 231 generates a synthetic sound (ensemble sound) by mixing the respective performance sounds acquired from the acquisition unit 230. The synthesis unit 231 generates the synthetic sound, for example, by compressing each sound source and adding the compressed sound sources. The synthesis unit 231 outputs the generated synthetic sound to the distribution unit 232.

[0054] The distribution unit 232 distributes the synthesized sound obtained from the synthesis unit 231 to the viewer terminal 30.

[0055] The viewer terminal 30 is a computer device of a viewer. The viewer terminal 30 is, for example, a smartphone, a PC, a tablet terminal, etc. The viewer terminal 30 includes, for example, a communication unit 31, a storage unit 32, a control unit 33, a display unit 34, and a speaker unit 35.

[0056] The communication unit 31 communicates with the distribution server 20. The storage unit 32 is configured with a storage medium such as a HDD, a flash memory, an EEPROM, a RAM, a ROM, or a combination of these. The storage unit 32 stores programs for executing various processes of the viewer terminal 30 and temporary data used when performing various processes.

[0057] The control unit 33 is realized by causing a CPU provided as hardware in the viewer terminal 30 to execute a program. The control unit 33 comprehensively controls the viewer terminal 30. The control unit 33 controls each of the communication unit 31, the storage unit 32, the display unit 34, and the speaker unit 35.

[0058] The display unit 34 includes a display device such as a liquid crystal display, and displays images such as video of a live performance related to the remote ensemble according to the control of the control unit 33.

[0059] The speaker unit 35 outputs ensemble sounds of a live performance related to the remote ensemble in response to the control of the control unit 33.

[0060] 6 is a sequence diagram illustrating the flow of processing performed by the ensemble system 1 according to the embodiment. In the following sequence diagram, a case will be described as an example in which two performer terminals 10-1 and 10-2 perform remote performance.

[0061] Performer terminal 10-1 collects the performance sounds in its own venue, and transmits the collected performance sounds to performer terminal 10-2 and distribution server 20 (step S10). The "own venue" here is the venue in which performer terminal 10-1 is installed.

[0062] Performer terminal 10-2 receives the performance sound from the other venue and performs sound processing of the received performance sound from the other venue (step S11). The other venue in this case is the venue where performer terminal 10-1 is installed. The flow of sound processing will be explained in detail later. Meanwhile, performer terminal 10-2 picks up the performance sound from its own venue and transmits the picked up performance sound to performer terminal 10-1 and distribution server 20 (step S12). The own venue in this case is the venue where performer terminal 10-2 is installed. Performer terminal 10-2 repeatedly executes the processes shown in steps S11 and S12 until the session ends.

[0063] The performer terminal 10-1 receives the performance sound from the other venue and performs sound processing of the received performance sound from the other venue (step S13). The other venue here is the venue where the performer terminal 10-2 is installed. The performer terminal 10-1 repeatedly executes the processes shown in steps S10 and S13 until the session ends.

[0064] The distribution server 20 receives the performance sound from the first venue (step S14). The first venue here is the venue where the performer terminal 10-1 is installed. The distribution server 20 also receives the performance sound from the second venue (step S15). The second venue here is the venue where the performer terminal 10-2 is installed. The distribution server 20 mixes the performance sound from the first venue and the performance sound from the second venue (step S16). The distribution server 20 transmits the mixed ensemble sound to the viewer terminal 30 (step S17). The viewer terminal 30 receives the ensemble sound distributed from the distribution server 20, and outputs the received ensemble sound to the speaker unit 35 for playback (step S18).

[0065] 7 is a flow chart for explaining the flow of sound processing performed by the performer terminal 10 according to the embodiment. The performer terminal 10 receives performance sounds from another venue (step S20). The performer terminal 10 estimates the performance sounds at a performance position (T+Δt) that is advanced by a time Δt from the performance position T of the received performance sounds (step S21). The performer terminal 10 outputs the estimated performance sounds from the speaker unit 15 (step S22). The performer terminal 10 collects the performance sounds from its own venue using the microphone unit 16 (step S23). The performer terminal 10 transmits the performance sounds collected at its own venue to the performer terminal 10 of the session partner and to the distribution server 20 (step S24).

[0066] As described above, the performer terminal 10 of the embodiment is provided in venue E1 when a remote ensemble is performed between venues E1 and E2. The performer terminal 10 includes an estimation unit 131. The estimation unit 131 estimates a future estimated performance sound from a performance sound. The performance sound is a sound picked up by a device (e.g., performer terminal 10-2) provided in venue E2. The estimation unit 131 inputs the performance sound to a performance sound estimation model to estimate an estimated performance sound. The performance sound estimation model is a trained model that estimates an estimated performance sound from the input performance sound. The performance sound estimation model is a trained model that has learned a sound signal corresponding to the performance sound.

[0067] Here, the performer terminal 10 is an example of a "device." In the above embodiment, a case has been described in which the performer terminal 10 installed in venue E estimates and outputs the performance sounds of other venues. However, this is not limited to this. At least any device installed in venue E may be configured to estimate and output the performance sounds of other venues. The device installed in venue E is, for example, a server device for distribution that distributes ensemble sounds, or a computer device such as a mixer that mixes the sounds of each venue.

[0068] Moreover, the ensemble system 1 of the embodiment includes performer terminals 10-1 and 10-2. Performer terminal 10-1 is provided in venue E1. Performer terminal 10-2 is provided in venue E2. Performer terminal 10 includes an acquisition unit 130, a communication unit 11, an estimation unit 131, and an output unit 132. The acquisition unit 130 of performer terminal 10-1 acquires a first performance sound in venue E1. The communication unit 11 of performer terminal 10-1 transmits the first performance sound to performer terminal 10-2. The communication unit 11 of performer terminal 10-1 receives a second performance sound in venue E2 from performer terminal 10-2. The estimation unit 131 of performer terminal 10-1 estimates a future performance sound (a second estimated performance sound) for the second performance sound received by the communication unit 11. The estimation unit 131 performs the estimation using the trained model (second performance sound estimation model). The output unit 132 of the performer terminal 10-1 outputs the estimated sound.

[0069] The acquisition unit 130 of the performer terminal 10-2 acquires a second performance sound at the venue E2. The communication unit 11 of the performer terminal 10-2 transmits the second performance sound to the performer terminal 10-1. The communication unit 11 of the performer terminal 10-2 receives the first performance sound from the performer terminal 10-1. The estimation unit 131 of the performer terminal 10-2 estimates a future performance sound (first estimated performance sound) for the first performance sound received by the communication unit 11. The estimation unit 131 performs the estimation using a trained model (first performance sound estimation model). The output unit 132 of the performer terminal 10-2 outputs the estimated sound.

[0070] The trained model (first performance sound estimation model) is a model that has trained a sound signal related to a performance sound (first performance sound). The trained model (second performance sound estimation model) is a model that has trained a sound signal related to a performance sound (second performance sound). As a result, in the ensemble system 1 of the embodiment, future performance sounds from performance sounds received via the communication network NW can be estimated and output. Therefore, even if a performance sound at a performance position T delayed from the actual performance position (T+Δt) is received due to a transmission delay, it is possible to estimate and output the performance sound at the actual performance position (T+Δt). Therefore, it is possible to reproduce the sound received via the communication line without delay.

[0071] In addition, in the performer terminal 10 of the embodiment, the trained model may be a model trained on a sound signal related to a rehearsal sound source. This allows the performance sound to be estimated with high accuracy.

[0072] In addition, a program for implementing the functions of the processing unit (control unit 13) in Fig. 1 may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed to perform construction management. Note that the term "computer system" here includes hardware such as the OS and peripheral devices.

[0073] In addition, if a WWW system is used, the "computer system" also includes the homepage providing environment (or display environment). Moreover, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. Furthermore, the term "computer-readable recording medium" refers to those that hold a program for a certain period of time, such as volatile memory inside a computer system that is a server or client. The above-mentioned program may be for realizing part of the above-mentioned functions, or may be capable of realizing the above-mentioned functions in combination with a program already recorded in the computer system. The above-mentioned program may be stored in a specified server, and the program may be distributed (downloaded, etc.) via a communication line in response to a request from another device.

[0074] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and designs that do not deviate from the gist of the present invention are also included. [Explanation of symbols]

[0075] REFERENCE SIGNS LIST 1 ensemble system, 10 performer terminal, 11 communication unit, 12 storage unit, 13 control unit, 14 display unit, 15 speaker unit, 16 microphone unit, 20 distribution server, 30 viewer terminal, 130 acquisition unit, 131 estimation unit, 132 output unit, 133 distribution unit

Claims

1. A device provided in a first venue when a remote ensemble is performed between a first venue and a second venue, an estimation unit that inputs the performance sound collected by the device installed in the second venue into a performance sound estimation model and estimates a future estimated performance sound for the performance sound; having the performance sound estimation model is a trained model trained to estimate the estimated performance sound from the input performance sound by learning a sound signal corresponding to the performance sound, Device.

2. the performance sound estimation model learns a rehearsal sound source corresponding to the performance sound; 2. The apparatus of claim 1.

3. An ensemble system for realizing a remote ensemble performance in a first venue and a second venue, comprising: a first terminal device provided in the first venue; and a second terminal device provided in the second venue; The first terminal device is a first acquisition unit that acquires a first performance sound in the first venue; a first transmitting unit for transmitting the first performance sound to the second terminal device; a first receiving unit that receives the second performance sound in the second venue from the second terminal device; a first estimating unit that estimates a future second estimated performance sound based on the second performance sound by inputting the second performance sound received by the first receiving unit to a second performance sound estimation model; a first sound output unit that outputs the second estimated musical performance sound; having The second terminal device is a second acquisition unit that acquires the second performance sound; a second transmitting unit for transmitting the second performance sound to the first terminal device; a second receiving unit for receiving the first performance sound from the first terminal device; a second estimation unit that estimates a future first estimated performance sound based on the first performance sound by inputting the first performance sound received by the second receiving unit to a first performance sound estimation model; a second sound output unit that outputs the first estimated musical performance sound; having the first performance sound estimation model is a trained model trained to estimate the first estimated performance sound from the input first performance sound by learning a first sound signal corresponding to the first performance sound, the second performance sound estimation model is a trained model trained to estimate the second estimated performance sound from the input second performance sound by learning a second sound signal corresponding to the second performance sound; Ensemble system.

4. A sound reproduction method performed by a computer device installed in a first venue when a remote ensemble is performed between a first venue and a second venue, comprising: A performance sound collected by a device installed in the second venue is input to a performance sound estimation model, and a future estimated performance sound for the performance sound is estimated; the performance sound estimation model is a trained model trained to estimate the estimated performance sound from the input performance sound by learning a sound signal corresponding to the performance sound, How to play sound.

5. The performance sound estimation model learns a rehearsal sound source corresponding to the performance sound.

5. The sound reproducing method according to claim 4.

6. In a case where a remote ensemble is performed between a first venue and a second venue, a computer device provided in the first venue, The performance sound collected by the device installed in the second venue is input to a performance sound estimation model, and a future estimated performance sound for the performance sound is estimated. A program, the performance sound estimation model is a trained model trained to estimate the estimated performance sound from the input performance sound by learning a sound signal corresponding to the performance sound, program.

7. The performance sound estimation model learns a rehearsal sound source corresponding to the performance sound. The program according to claim 6.

Images