Live data distribution method, live data distribution system, live data distribution device, live data playback device, and live data playback method

Abstract

To provide, when distributing live data, realism of a live venue to a venue of a distribution destination.SOLUTION: A live data distribution method distributes, as distribution data, sound source information related to sound generated in a first venue and spatial echo information that is information related to initial reflection sound changing according to a position of the sound, renders the distribution data to provide the sound related to the sound source information and the sound related to the spatial echo, to a second venue. The spatial echo information includes information for generating indirect sound. The rendering includes processing of generating indirect sound of sound from the sound source. The processing of generating the indirect sound includes processing of generating the initial reflection sound. The initial reflection sound to be generated changes according to a position of the sound. The spatial echo information further includes information on rear reverberation sound not changing according to the position of the sound. The processing of generating the indirect sound further includes processing of generating the rear reverberation sound.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] An embodiment of the present invention relates to a live data distribution method, a live data distribution system, a live data distribution device, a live data playback device, and a live data playback method. [Background technology]

[0002] Patent Document 1 discloses a system for rendering spatial audio content in a listening environment to provide a more immersive spatial listening experience.

[0003] The system of Patent Document 1 is described as measuring the impulse response of a sound output from a speaker in a listening environment, and performing a filter process according to the measured impulse response. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Publication No. 2015-530043 Summary of the Invention [Problem to be solved by the invention]

[0005] The system of Patent Document 1 is not a live data distribution system. When distributing live data, it is desirable to provide the venue at the distribution destination with the same sense of realism as the live venue.

[0006] An embodiment of the present invention aims to provide a live data distribution method, a live data distribution system, a live data distribution device, a live data playback device, and a live data playback method that can provide the same sense of realism as a live venue at a distribution destination venue when distributing live data. [Means for solving the problem]

[0007] A live data distribution method includes distributing, as distribution data, sound source information relating to a sound generated in a first venue and spatial reverberation information, which is information relating to early reflection sounds that change depending on the position of the sound, and rendering the distribution data to provide the sound relating to the sound source information and the sound relating to the spatial reverberation to a second venue. The spatial reverberation information includes information for generating indirect sound, the rendering includes a process of generating indirect sound of the sound source, the process of generating the indirect sound includes a process of generating early reflection sounds, the generated early reflection sounds change depending on the position of the sound, the spatial reverberation information further includes information about rear reverberation sounds that do not change depending on the position of the sound, and the process of generating the indirect sound further includes a process of generating the rear reverberation sounds. Effect of the Invention

[0008] The live data distribution method can provide the venue, which is the distribution destination, with the same sense of realism as the live venue when distributing live data. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing a configuration of a live data distribution system 1. [Diagram 2] FIG. 2 is a schematic plan view of the first venue 10. [Diagram 3] FIG. 2 is a schematic plan view of the second venue 20. [Figure 4] FIG. 2 is a block diagram showing a configuration of a mixer 11. [Diagram 5] 2 is a block diagram showing a configuration of a distribution device 12. FIG. [Figure 6] 4 is a flowchart showing the operation of the distribution device 12. [Figure 7] FIG. 2 is a block diagram showing a configuration of a playback device 22. [Figure 8] 13 is a flowchart showing the operation of the playback device 22. [Figure 9] FIG. 1 is a block diagram showing a configuration of a live data distribution system 1A according to a first modified example. [Figure 10]1 is a schematic plan view of a second venue 20 in a live data distribution system 1A according to a first modified example. FIG. [Figure 11] FIG. 11 is a block diagram showing a configuration of a live data distribution system 1B according to a second modified example. [Figure 12] FIG. 2 is a block diagram showing the configuration of an AV receiver 32. [Figure 13] FIG. 11 is a block diagram showing a configuration of a live data distribution system 1C according to a third modified example. [Figure 14] FIG. 4 is a block diagram showing the configuration of a terminal 42. [Figure 15] FIG. 13 is a block diagram showing a configuration of a live data distribution system 1D according to a fourth modified example. [Figure 16] FIG. 7 is a diagram showing an example of live video 700 displayed on a playback device in each venue. [Figure 17] FIG. 11 is a block diagram showing an application example of signal processing performed in a playback device. [Figure 18] 1 is a schematic diagram showing the path of sound that travels from a sound source 70, reflects off a wall surface, and arrives at a sound receiving point 75. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] 1 is a block diagram showing the configuration of a live data distribution system 1. The live data distribution system 1 is made up of a plurality of audio devices and information processing devices installed in a first venue 10 and a second venue 20, respectively.

[0011] Fig. 2 is a schematic plan view of the first venue 10, and Fig. 3 is a schematic plan view of the second venue 20. In this example, the first venue 10 is a live venue where performers perform. The second venue 20 is a public viewing venue where listeners in remote locations watch the performers' performance.

[0012] A mixer 11, a distribution device 12, a plurality of microphones 13A to 13F, a plurality of speakers 14A to 14G, a plurality of trackers 15A to 15C, and a camera 16 are installed in the first venue 10. A mixer 21, a playback device 22, a display 23, and a plurality of speakers 24A to 24F are installed in the second venue 20. The distribution device 12 and the playback device 22 are connected via the Internet 5. Note that the number of microphones, the number of speakers, the number of trackers, and the like are not limited to the numbers shown in this embodiment. Furthermore, the installation manner of the microphones and speakers is not limited to the example shown in this embodiment.

[0013] The mixer 11 is connected to a distribution device 12, a plurality of microphones 13A to 13F, a plurality of speakers 14A to 14G, and a plurality of trackers 15A to 15C. The mixer 11, a plurality of microphones 13A to 13F, and a plurality of speakers 14A to 14G are connected via a network cable or an audio cable. The plurality of trackers 15A to 15C are connected to the mixer 11 via wireless communication. The mixer 11 and the distribution device 12 are connected via a network cable. In addition, the distribution device 12 is connected to a camera 16 via a video cable. The camera 16 captures live video including the performers.

[0014] A plurality of speakers 14A to 14G are installed along the walls of the first venue 10. In this example, the first venue 10 is rectangular in plan view. A stage is arranged in front of the first venue 10. On the stage, performers perform singing or musical instruments. Speaker 14A is installed on the left side of the stage, speaker 14B is installed in the center of the stage, and speaker 14C is installed on the right side of the stage. Speaker 14D is installed on the left side of the center of the front and rear of the first venue 10, and speaker 14E is installed on the right side of the center of the front and rear of the first venue 10. Speaker 14F is installed on the left side of the rear of the first venue 10, and speaker 14G is installed on the right side of the rear of the first venue 10.

[0015] Microphone 13A is installed on the left side of the stage, microphone 13B is installed in the center of the stage, and microphone 13C is installed on the right side of the stage. Microphone 13D is installed on the left side of the center of the front and rear of first venue 10, and microphone 13E is installed in the center of the rear of first venue 10. Microphone 13F is installed on the right side of the center of the front and rear of first venue 10.

[0016] Mixer 11 receives sound signals from microphones 13A to 13F. Mixer 11 also outputs sound signals to speakers 14A to 14G. In this embodiment, speakers and microphones are shown as examples of audio devices connected to mixer 11, but in reality, a large number of audio devices are connected to mixer 11. Mixer 11 receives sound signals from multiple audio devices such as microphones, performs signal processing such as mixing, and outputs sound signals to multiple audio devices such as speakers.

[0017] The microphones 13A to 13F each capture the singing or playing sound of the performer as a sound generated in the first venue 10. Alternatively, the microphones 13A to 13F capture the environmental sound of the first venue 10. In the example of FIG. 2, the microphones 13A to 13C capture the performer's sound, and the microphones 13D to 13F capture the environmental sound. The environmental sound includes the listeners' cheers, applause, calls, cheers, chorus, or murmurs. However, the performer's sound may be line-input. Line input means inputting a sound signal from an audio cable or the like connected to the sound source, rather than picking up the sound output from a sound source such as an instrument and inputting it with a microphone. It is preferable that the performer's sound is captured as a sound with a high signal-to-noise ratio and does not include other sounds.

[0018] The speakers 14A to 14G output the sounds of the performers to the first venue 10. The speakers 14A to 14G may also output early reflected sounds or late reverberation sounds for controlling the sound field in the first venue 10.

[0019] The mixer 21 in the second venue 20 is connected to a playback device 22 and a number of speakers 24A to 24F. These audio devices are connected via a network cable or an audio cable. The playback device 22 is connected to a display 23 via a video cable.

[0020] A plurality of speakers 24A to 24F are installed along the walls of the second venue 20. In this example, the second venue 20 is rectangular in plan view. A display 23 is disposed in front of the second venue 20. The display 23 displays live video captured in the first venue 10. The speaker 24A is installed on the left side of the display 23, and the speaker 24B is installed on the right side of the display 23. The speaker 24C is installed on the left side of the center between the front and rear of the second venue 20, and the speaker 24D is installed on the right side of the center between the front and rear of the second venue 20. The speaker 24E is installed on the left side at the rear of the second venue 20, and the speaker 24F is installed on the right side at the rear of the second venue 20.

[0021] The mixer 21 outputs audio signals to the speakers 24A to 24F. The mixer 21 receives audio signals from the playback device 22, performs signal processing such as mixing, and outputs the audio signals to a plurality of audio devices such as speakers.

[0022] The speakers 24A to 24F output the sounds of the performers to the second venue 20. The speakers 24A to 24F also output early reflection sounds or late reverberation sounds to reproduce the sound field of the first venue 10. The speakers 24A to 24F also output environmental sounds, such as cheers from the listeners in the first venue 10, to the second venue 20.

[0023] Fig. 4 is a block diagram showing the configuration of mixer 11. Since mixer 21 has the same configuration and functions as mixer 11, Fig. 4 shows the configuration of mixer 11 as a representative. Mixer 11 includes a display 101, a user I / F 102, an audio I / O (Input / Output) 103, a signal processing unit (DSP) 104, a network I / F 105, a CPU 106, a flash memory 107, and a RAM 108.

[0024] The CPU 106 is a control unit that controls the operation of the mixer 11. The CPU 106 performs various operations by reading out a predetermined program stored in a flash memory 107, which is a storage medium, into a RAM 108 and executing it.

[0025] It should be noted that the program read by CPU 106 does not need to be stored in flash memory 107 in the device itself. For example, the program may be stored in a storage medium of an external device such as a server. In this case, CPU 106 may read the program from the server to RAM 108 each time and execute it.

[0026] The signal processing unit 104 is composed of a DSP for performing various signal processing. The signal processing unit 104 performs signal processing such as mixing and filtering on an audio signal input from an audio device such as a microphone via the audio I / O 103 or the network I / F 105. The signal processing unit 104 outputs the processed audio signal to an audio device such as a speaker via the audio I / O 103 or the network I / F 105.

[0027] The signal processor 104 may also perform panning, early reflection sound generation, and late reverberation sound generation. The panning process is a process for controlling the volume of sound signals distributed to the multiple speakers 14A-14G so that a sound image is localized at the position of the performer. To perform the panning process, the CPU 106 acquires the position information of the performer via the trackers 15A-15C. The position information is information indicating two-dimensional or three-dimensional coordinates based on a certain position of the first venue 10. The trackers 15A-15C are tags that transmit and receive radio waves, such as Bluetooth (registered trademark). The performers or musical instruments are fitted with the trackers 15A-15C. At least three beacons are installed in advance in the first venue 10. Each beacon measures the distance to the trackers 15A-15C based on the time difference between transmitting and receiving radio waves. The CPU 106 acquires the position information of the beacons in advance and measures the distances from at least three beacons to the tag, thereby making it possible to uniquely determine the positions of the trackers 15A to 15C.

[0028] In this way, the CPU 106 acquires position information of each performer, that is, position information of the sound generated in the first venue 10, via the trackers 15A-15C. Based on the acquired position information and the positions of the speakers 14A-14G, the CPU 106 determines the volume of each sound signal to be output to the speakers 14A-14G so that the sound image is localized at the position of the performer. The signal processor 104 controls the volume of each sound signal to be output to the speakers 14A-14G under the control of the CPU 106. For example, the signal processor 104 increases the volume of the sound signal to be output to the speaker close to the position of the performer and decreases the volume of the sound signal to be output to the speaker far from the position of the performer. In this way, the signal processor 104 can localize the sound image of the performance sound or singing sound of the performer at a predetermined position.

[0029] The early reflection sound generation process and the late reverberation sound generation process are processes that convolve an impulse response with the performer's sound using an FIR filter. The signal processing unit 104 convolves the performer's sound with an impulse response acquired in advance, for example, at a specific venue (a venue other than the first venue 10). In this way, the signal processing unit 104 controls the sound field of the first venue 10. Alternatively, the signal processing unit 104 may control the sound field of the first venue 10 by further feeding back sounds acquired by microphones installed near the ceiling or walls of the first venue 10 to the speakers 14A to 14G.

[0030] The signal processing unit 104 outputs the sound of the performer and the information on the performer's position to the distribution device 12. The distribution device 12 acquires the sound of the performer and the information on the performer's position from the mixer 11.

[0031] In addition, the distribution device 12 obtains a video signal from the camera 16. The camera 16 captures images of each performer or the entire first venue 10, and outputs a video signal related to the live image to the distribution device 12.

[0032] Furthermore, the distribution device 12 acquires spatial reverberation information of the first venue 10. The spatial reverberation information is information for generating indirect sound. Indirect sound is sound that the sound of the sound source is reflected in the venue and reaches the listener, and includes at least early reflection sound and late reverberation sound. The spatial reverberation information includes, for example, information indicating the size, shape, and material of the walls of the first venue 10, and an impulse response related to the late reverberation sound. The information indicating the size, shape, and material of the walls of the space is information for generating early reflection sound. The information for generating early reflection sound may be an impulse response. The impulse response is measured in advance, for example, in the first venue 10. The spatial reverberation information may also be information that changes depending on the position of the performer. The information that changes depending on the position of the performer is, for example, an impulse response measured in advance for each position of the performer in the first venue 10. For example, the distribution device 12 obtains a first impulse response when a performer's sound occurs in front of the stage of the first venue 10, a second impulse response when a performer's sound occurs on the left side of the stage, and a third impulse response when a performer's sound occurs on the right side of the stage. However, the number of impulse responses is not limited to three. In addition, the impulse responses do not need to be actually measured in the first venue 10, and may be obtained, for example, by simulation based on the size, shape, and material of the walls of the space of the first venue 10.

[0033] The early reflection sound is a reflection sound with a fixed direction of arrival, while the late reverberation sound is a reflection sound with an uncertain direction of arrival. Late reverberation sound changes less with changes in the position of the performer than the early reflection sound. Therefore, the reverberation information of the space may be composed of an impulse response of the early reflection sound that changes depending on the position of the performer, and an impulse response of the late reverberation sound that is constant regardless of the position of the performer.

[0034] Furthermore, the signal processing unit 104 may obtain ambience information related to the environmental sound and output it to the distribution device 12. The environmental sound is the sound obtained by the microphones 13D to 13F as described above, and includes sounds such as background noise, cheers from listeners, applause, calls, cheers, chorus, or murmurs. However, the environmental sound may be obtained by the microphones 13A to 13C on the stage. The signal processing unit 104 outputs a sound signal related to the environmental sound to the distribution device 12 as ambience information. The ambience information may include position information of the environmental sound. Among the environmental sounds, cheers such as "Go for it!" from each listener, calls on the names of the performers, or exclamations such as "Bravo" are sounds that can be recognized as the voices of individual listeners without being drowned out by the audience. The signal processing unit 104 may obtain position information of these individual sounds. The position information of the environmental sound can be obtained, for example, from the sounds obtained by the microphones 13D to 13F. When the signal processing unit 104 recognizes the individual sounds by processing such as voice recognition, it finds the correlation between the sound signals of the microphones 13D to 13F and finds the difference in timing at which the individual sounds are picked up by the microphones 13D to 13F. Based on the difference in timing at which the sounds are picked up by the microphones 13D to 13F, the signal processing unit 104 can uniquely find the position in the first venue 10 where the sound originated. Furthermore, the position information of the environmental sound may be regarded as the position information of each of the microphones 13D to 13F.

[0035] The distribution device 12 encodes and distributes sound source information related to sounds generated in the first venue 10 and spatial reverberation information as distribution data. The sound source information includes at least the sounds of the performers, but may also include position information of the sounds of the performers. The distribution device 12 may also distribute ambience information related to environmental sounds included in the distribution data. The distribution device 12 may also distribute a video signal related to the video of the performers included in the distribution data.

[0036] Alternatively, the distribution device 12 may distribute at least sound source information related to the sound of the performer and position information of the performer, and ambience information related to environmental sound as distribution data.

[0037] Fig. 5 is a block diagram showing the configuration of the distribution device 12. Fig. 6 is a flowchart showing the operation of the distribution device 12.

[0038] The distribution device 12 is an information processing device such as a general personal computer, etc. The distribution device 12 includes a display 201, a user I / F 202, a CPU 203, a RAM 204, a network I / F 205, a flash memory 206, and a general-purpose communication I / F 207.

[0039] CPU 203 reads out a program stored in flash memory 206, which is a storage medium, into RAM 204 to realize a predetermined function. Note that the program read out by CPU 203 does not need to be stored in flash memory 206 in its own device. For example, the program may be stored in a storage medium of an external device such as a server. In this case, CPU 203 may read the program from the server into RAM 204 each time and execute it.

[0040] The CPU 203 acquires the sound of the performers and information on the performers' positions (sound source information) from the mixer 11 via the network I / F 205 (S11). The CPU 203 also acquires reverberation information for the space of the first venue 10 (S12). The CPU 203 further acquires ambience information relating to environmental sounds (S13). The CPU 203 may also acquire a video signal from the camera 16 via the general-purpose communication I / F 207.

[0041] The CPU 203 encodes and distributes data relating to the performer's sound and sound position information (sound source information), data relating to spatial reverberation information, data relating to ambience information, and data relating to video signals as distribution data (S14).

[0042] The playback device 22 receives distribution data from the distribution device 12 via the Internet 5. The playback device 22 renders the distribution data and provides the second venue 20 with sounds of the performers and sounds related to the reverberation of the space. Alternatively, the playback device 22 provides the second venue 20 with sounds of the performers and environmental sounds included in the ambience information. The playback device 22 may provide the second venue 20 with sounds related to the reverberation of the space corresponding to the ambience information.

[0043] Fig. 7 is a block diagram showing the configuration of the playback device 22. Fig. 8 is a flowchart showing the operation of the playback device 22.

[0044] The playback device 22 is an information processing device such as a general personal computer, etc. The playback device 22 includes a display 301, a user I / F 302, a CPU 303, a RAM 304, a network I / F 305, a flash memory 306, and a video I / F 307.

[0045] CPU 303 reads out a program stored in flash memory 306, which is a storage medium, into RAM 304 to realize a predetermined function. Note that the program read out by CPU 303 does not need to be stored in flash memory 306 in its own device. For example, the program may be stored in a storage medium of an external device such as a server. In this case, CPU 303 may read the program from the server into RAM 304 each time and execute it.

[0046] The CPU 303 receives distribution data from the distribution device 12 via the network I / F 305 (S21). The CPU 303 decodes the distribution data into sound source information, spatial reverberation information, ambience information, video signals, etc. (S22), and renders the sound source information, spatial reverberation information, ambience information, video signals, etc.

[0047] As an example of rendering the sound source information, the CPU 303 causes the mixer 21 to perform panning processing of the performer's sound (S23). The panning processing is processing for localizing the performer's sound to the performer's position as described above. The CPU 303 determines the volume of the sound signal to be distributed to the speakers 24A-24F so that the performer's sound is localized to the position indicated by the position information included in the sound source information. The CPU 303 causes the mixer 21 to perform panning processing by outputting to the mixer 21 the sound signal related to the performer's sound and information indicating the output amount of the sound signal related to the performer's sound to the speakers 24A-24F.

[0048] This allows the listeners in the second venue 20 to perceive the sound as coming from the performer's position. The listeners in the second venue 20 can hear the sound of a performer on the right side of the stage in the first venue 10 from the front right side in the second venue 20. The CPU 303 may also render the video signal and display live video on the display 23 via the video I / F 307. This allows the listeners in the second venue 20 to hear the sound of the performer that has been panned while watching the video of the performer displayed on the display 23. This allows the listeners in the second venue 20 to feel more immersed in the live performance because the visual information and the auditory information match.

[0049] Furthermore, the CPU 303 causes the mixer 21 to perform an indirect sound generation process as an example of rendering the reverberation information of the space (S24). The indirect sound generation process includes an early reflection sound generation process and a late reverberation sound generation process. The early reflection sound is generated based on the sound of the performer contained in the sound source information and information indicating the size, shape, and material of the wall of the first venue 10 contained in the reverberation information of the space. The CPU 303 determines the arrival timing of the early reflection sound based on the size and shape of the space, and determines the level of the early reflection sound based on the material of the wall. More specifically, the CPU 303 obtains the coordinates of the wall surface on which the sound of the sound source is reflected based on the information on the size and shape of the space. Then, the CPU 303 obtains the position of a virtual sound source (imaginary sound source) that exists with the wall surface as a mirror surface with respect to the position of the sound source based on the position of the sound source, the position of the wall surface, and the position of the sound receiving point. The CPU 303 obtains the delay amount of the imaginary sound source based on the distance from the position of the imaginary sound source to the sound receiving point. The CPU 303 also obtains the level of the imaginary sound source based on the information on the material of the wall. The information on the material corresponds to the energy loss when reflected by the wall. Therefore, the CPU 303 obtains the level of the imaginary sound source by considering the energy loss in the sound signal of the sound source. By repeating such processing, the CPU 303 can obtain the delay amount and level of the sound related to the reverberation of the space by calculation. The CPU 303 outputs the calculated delay amount and level to the mixer 21. The mixer 21 convolves the level tap coefficient corresponding to the delay amount and level into the sound of the performer. In this way, the mixer 21 reproduces the reverberation of the space of the first venue 10 in the second venue 20. Also, if the reverberation information of the space includes the impulse response of the early reflection sound, the CPU 303 causes the mixer 11 to execute a process of convolving the impulse response into the sound of the performer using an FIR filter. The CPU 303 outputs the reverberation information of the space (impulse response) included in the distribution data to the mixer 21. The mixer 21 convolves the spatial reverberation information (impulse response) received from the playback device 22 with the sound of the performer. In this way, the mixer 21 reproduces the spatial reverberation of the first venue 10 in the second venue 20.

[0050] Furthermore, when the spatial reverberation information changes according to the position of the performer, the playback device 22 outputs the spatial reverberation information corresponding to the performer's position to the mixer 21 based on the position information included in the sound source information. For example, when a performer who was in front of the stage in the first venue 10 moves to the left side of the stage, the impulse response to be convoluted with the performer's sound is changed from the first impulse response to the second impulse response. Alternatively, when reproducing a virtual sound source based on information on the size and shape of the space, the delay amount and level are recalculated according to the position of the performer after the movement. In this way, the appropriate spatial reverberation according to the performer's position is also reproduced in the second venue 20.

[0051] The reproduction device 22 may also cause the mixer 21 to generate a spatial reverberation sound corresponding to the environmental sound based on the ambience information and the spatial reverberation information. In other words, the sound related to the spatial reverberation may include a first reverberation sound corresponding to the performer's sound (sound of the first sound source) and a second reverberation sound corresponding to the environmental sound (sound of the second sound source). In this way, the mixer 21 reproduces the reverberation of the environmental sound in the first venue 10 in the second venue 20. In addition, if the ambience information includes position information, the reproduction device 22 may output the spatial reverberation information corresponding to the position of the environmental sound to the mixer 11 based on the position information included in the ambience information. The mixer 21 reproduces the reverberation of the environmental sound based on the position of the environmental sound. For example, if a spectator who was at the left rear of the first venue 10 moves to the right rear, the impulse response to be convoluted with the cheers of that spectator is changed. Alternatively, when reproducing a virtual sound source based on information on the size and shape of a space, the delay amount and level are recalculated according to the position of the audience after they have moved. In this way, the reverb information of the space may include first reverb information that changes according to the position of the performer's sound (first sound source) and second reverb information that changes according to the position of the environmental sound (second sound source), and the rendering may include a process of generating the first reverb based on the first reverb information and a process of generating the second reverb based on the second reverb information.

[0052] Furthermore, late reverberation sounds are reflected sounds that come from an uncertain direction. Late reverberation sounds change less with changes in sound position than early reflected sounds. Therefore, the playback device 22 may change only the impulse response of the early reflected sounds, which change depending on the position of the performer, and keep the impulse response of the late reverberation sounds fixed.

[0053] The playback device 22 may omit the indirect sound generation process and use the reverberation of the second venue 20 as it is. The indirect sound generation process may be only the early reflection sound generation process. The rear reverberation may use the reverberation of the second venue 20 as it is. Alternatively, the mixer 21 may reinforce the control of the second venue 20 by further feeding back to the speakers 24A to 24F the sound picked up by a microphone (not shown) installed near the ceiling or wall of the second venue 20.

[0054] Then, the CPU 303 of the playback device 22 performs playback processing of the environmental sound based on the ambience information (S25). The ambience information includes sound signals of sounds such as background noise, cheers from listeners, applause, calls, cheers, chorus, and murmurs. The CPU 303 outputs these sound signals to the mixer 21. The mixer 21 outputs the sound signals received from the playback device 22 to the speakers 24A to 24F.

[0055] When the ambience information includes position information of the environmental sound, the CPU 303 causes the mixer 21 to perform localization processing of the environmental sound by panning processing. In this case, the CPU 303 determines the volume of the sound signal to be distributed to the speakers 24A to 24F so that the environmental sound is localized at the position of the position information included in the ambience information. The CPU 303 causes the mixer 21 to perform panning processing by outputting the sound signal of the environmental sound and information indicating the output amount of the sound signal related to the environmental sound to the speakers 24A to 24F to the mixer 21. The same is true when the position information of the environmental sound is the position information of each of the microphones 13D to 13F. The CPU 303 determines the volume of the sound signal to be distributed to the speakers 24A to 24F so that the environmental sound is localized at the position of the microphone. Each of the microphones 13D to 13F collects a plurality of environmental sounds (second sound sources) such as background noise, applause, chorus, cheers such as "Wow", and murmurs. The sound of each sound source reaches the microphone with a predetermined delay and level. In other words, background noise, applause, chorus, cheers such as "Wow!", and chatter reach the microphone as individual sound sources with a predetermined delay and level (information for localizing the sound source). The CPU 303 performs a panning process so that the sound picked up by the microphone is localized to the position of the microphone, thereby easily reproducing the localization of each sound source.

[0056] For sounds that cannot be recognized as the voices of individual listeners and are emitted simultaneously by many listeners, the CPU 303 may perform processing to make the listener perceive a spatial spread by having the mixer 21 perform effect processing such as reverb. For example, background noise, applause, chorus, cheers such as "Wow!", and murmuring are sounds that resonate throughout a live music venue. The CPU 303 causes the mixer 21 to perform effect processing to make the listener perceive a spatial spread for these sounds.

[0057] The playback device 22 may provide environmental sounds based on the above-mentioned ambience information to the second venue 20. This allows the listeners in the second venue 20 to enjoy a more realistic live performance, as if they were watching a live performance in the first venue 10.

[0058] In this manner, the live data distribution system 1 of this embodiment distributes sound source information related to sounds generated in the first venue 10 and spatial reverberation information as distribution data, renders the distribution data, and provides sounds related to the sound source information and sounds related to spatial reverberation to the second venue 20. This makes it possible to provide the sense of realism of the live venue to the distribution destination venue as well.

[0059] Furthermore, the live data distribution system 1 distributes, as distribution data, the sound of a first sound source (e.g., the sound of a performer) generated at a first location (e.g., a stage) in the first venue 10, first sound source information relating to position information of the first sound source, and second sound source information relating to a second sound source (e.g., environmental sound) generated at a second location (e.g., a location where a listener is) in the first venue 10, and renders the distribution data to provide the sound of the first sound source, which has been localized based on the position information of the first sound source, and the sound of the second sound source, to the second venue. This allows the sense of realism of the live venue to be provided to the venue to which the content is distributed.

[0060] Next, Fig. 9 is a block diagram showing the configuration of a live data distribution system 1A according to Modification 1. Fig. 10 is a schematic plan view of a second venue 20 in the live data distribution system 1A according to Modification 1. Configurations common to Figs. 1 and 3 are given the same reference numerals and descriptions thereof will be omitted.

[0061] A plurality of microphones 25A to 25C are installed in the second venue 20 of the live data distribution system 1A. When facing the stage 80 of the second venue 20, the microphone 25A is installed on the left side of the center in the front and rear, and the microphone 25B is installed in the center in the rear of the second venue 20. The microphone 25C is installed on the right side of the center in the front and rear of the second venue 20.

[0062] The microphones 25A to 25C acquire environmental sounds in the second venue 20. The mixer 21 outputs the sound signals of the environmental sounds as ambience information to the playback device 22. The ambience information may include position information of the environmental sounds. The position information of the environmental sounds can be obtained, for example, from the sounds acquired by the microphones 25A to 25C, as described above.

[0063] The playback device 22 transmits ambience information related to the environmental sound generated in the second venue 20 as a third sound source to the other venues. For example, the playback device 22 feeds back the environmental sound generated in the second venue 20 to the first venue 10. This allows the performers on the stage of the first venue 10 to hear voices, applause, cheers, etc. of people other than the listeners in the first venue 10, and allows the live performance to be performed in a realistic environment. Furthermore, the listeners in the first venue 10 can also hear the voices, applause, cheers, etc. of listeners in the other venues, and allows the live performance to be viewed in a realistic environment.

[0064] Furthermore, if a playback device at another venue renders the distribution data and provides the sound from the first venue to that other venue, and also provides the ambient sounds generated at the second venue 20 to that other venue, listeners at that other venue will be able to hear the voices, applause, cheers, etc. of many listeners, and will be able to watch the live performance in a more immersive environment.

[0065] Next, Fig. 11 is a block diagram showing the configuration of a live data distribution system 1B according to Modification 2. Configurations common to Fig. 1 are given the same reference numerals and description thereof will be omitted.

[0066] In the live data distribution system 1B, the distribution device 12 is connected to an AV receiver 32 in a third venue 20A via the Internet 5. The AV receiver 32 is connected to a display device 33, a plurality of speakers 34A-34F, and a microphone 35. The third venue 20A is, for example, the home of a listener. The AV receiver 32 is an example of a playback device. A user of the AV receiver 32 becomes a listener who remotely watches a live performance in the first venue 10.

[0067] 12 is a block diagram showing the configuration of the AV receiver 32. The AV receiver 32 includes a display 401, a user I / F 402, an audio I / O (Input / Output) 403, a signal processing unit (DSP) 404, a network I / F 405, a CPU 406, a flash memory 407, a RAM 408, and a video I / F 409.

[0068] The CPU 406 is a control unit that controls the operation of the AV receiver 32. The CPU 406 performs various operations by loading a predetermined program stored in a flash memory 407, which is a storage medium, into a RAM 408 and executing the program.

[0069] The programs read by CPU 406 do not need to be stored in flash memory 407 in the device itself. For example, the programs may be stored in a storage medium of an external device such as a server. In this case, CPU 406 only needs to read the programs from the server into RAM 408 and execute them.

[0070] The signal processing unit 404 is composed of a DSP for performing various signal processing. The signal processing unit 404 performs signal processing on an audio signal input via the audio I / O 403 or the network I / F 405. The signal processing unit 404 outputs the processed audio signal to an audio device such as a speaker via the audio I / O 403 or the network I / F 405.

[0071] The AV receiver 32 performs the same processing as that performed by the mixer 21 and the playback device 22. The CPU 406 receives distribution data from the distribution device 12 via the network I / F 405. The CPU 406 renders the distribution data and provides the performer's sound and the sound related to the reverberation of the space to the third venue 20A. Alternatively, the CPU 406 renders the distribution data and provides the environmental sound generated in the first venue 10 to the third venue 20A. Alternatively, the CPU 406 may render the distribution data and display live video on the display 33 via the video I / F 307.

[0072] The signal processing unit 404 performs panning processing of the performer's sound. The signal processing unit 404 also performs processing to generate indirect sound. Alternatively, the signal processing unit 404 may perform panning processing of environmental sound.

[0073] This allows the AV receiver 32 to provide the same sense of realism as the first venue 10 to the third venue 20A.

[0074] The AV receiver 32 also acquires environmental sounds of the third venue 20A (sounds such as cheers, applause, or calls from listeners) via the microphone 35. The AV receiver 32 transmits the environmental sounds of the third venue 20A to other devices. For example, the AV receiver 32 feeds back the environmental sounds of the third venue 20A to the first venue 10.

[0075] In this way, by feeding back sounds from multiple listeners to the first venue 10, the performers on stage at the first venue 10 can hear the cheers, applause, cheers, etc. of many listeners other than the listeners at the first venue 10, and can perform a live performance in a very realistic environment. Also, the listeners at the first venue 10 can hear the cheers, applause, cheers, etc. of many listeners in remote locations, and can watch the live performance in a very realistic environment.

[0076] Alternatively, the AV receiver 32 may receive reactions from listeners by displaying icon images such as "cheering," "applause," "calling out," and "bustling noise" on the display 401 and receiving a selection operation for these icon images from the listener via the user I / F 402. When the AV receiver 32 receives a selection operation for these reactions, it may generate sound signals corresponding to the respective reactions and transmit them to another device as ambience information.

[0077] Alternatively, the AV receiver 32 may transmit information indicating the type of environmental sound, such as the listener's cheers, applause, or calls, as ambience information. In this case, the receiving device (e.g., the distribution device 12 and the mixer 11) generates a corresponding sound signal based on the ambience information, and provides the listener's cheers, applause, calls, or other sounds in the venue. In this way, the ambience information is not a sound signal of the environmental sound, but information indicating the sound to be generated, and the distribution device 12 and the mixer 11 may play back pre-recorded environmental sounds, etc.

[0078] Furthermore, the ambience information of the first venue 10 may also be pre-recorded environmental sounds, rather than environmental sounds generated in the first venue 10. In this case, the distribution device 12 distributes information indicating sounds to be generated as the ambience information. The playback device 22 or the AV receiver 32 plays the corresponding environmental sounds based on the ambience information. Furthermore, among the ambience information, background noise and commotion may be recorded sounds, and other environmental sounds (e.g., cheers, applause, calls from listeners, etc.) may be sounds generated in the first venue 10.

[0079] The AV receiver 32 may also receive position information of the listener via the user I / F 402. The AV receiver 32 displays an image simulating a plan view or a perspective view of the first venue 10 on the display 401 or the display 33, and receives position information from the listener via the user I / F 402 (see FIG. 16, for example). The position information is information that specifies an arbitrary position within the first venue 10. The AV receiver 32 transmits the received position information of the listener to the first venue 10. The distribution device 12 and mixer 11 of the first venue perform a process of localizing the environmental sound of the third venue 20A to the specified position based on the environmental sound of the third venue 20A and the position information of the listener received from the AV receiver 32.

[0080] The AV receiver 32 may also change the content of the panning process based on position information received from the user. For example, if the listener specifies a position just in front of the stage in the first venue 10, the AV receiver 32 sets the localization position of the performer's sound to a position just in front of the listener and performs panning. This allows the listener in the third venue 20A to feel as if he or she is just in front of the stage in the first venue 10.

[0081] The sound of the listener in the third venue 20A may be transmitted to the second venue 20, not to the first venue 10, or may be transmitted to other venues. For example, the sound of the listener in the third venue 20A may be transmitted only to a friend's house (fourth venue). The listener in the fourth venue can watch the live performance in the first venue 10 while listening to the sound of the listener in the third venue 20A. In addition, a playback device (not shown) in the fourth venue may transmit the sound of the listener in the fourth venue to the third venue 20A. In this case, the listener in the third venue 20A can watch the live performance in the first venue 10 while listening to the sound of the listener in the fourth venue. This allows the listener in the third venue 20A and the listener in the fourth venue to watch the live performance in the first venue 10 while talking to each other.

[0082] Fig. 13 is a block diagram showing the configuration of a live data distribution system 1C according to Modification 3. Components common to Fig. 1 are given the same reference numerals and description thereof will be omitted.

[0083] In the live data distribution system 1C, the distribution device 12 is connected to a terminal 42 in a fifth venue 20B via the Internet 5. The terminal 42 is connected to headphones 43. The fifth venue 20B is, for example, a listener's home. However, if the terminal 42 is portable, the fifth venue 20B may be any place, such as a cafe, a car, or public transportation. In this case, any place can be the fifth venue 20B. The terminal 42 is an example of a playback device. The user of the terminal 42 is a listener who remotely watches the live performance in the first venue 10. In this case, the terminal 42 renders the distribution data and provides the sound related to the sound source information and the sound related to the reverberation of the space to the second venue (the fifth venue 20B in this example) via the headphones 43.

[0084] 14 is a block diagram showing the configuration of the terminal 42. The terminal 42 is an information processing device such as a personal computer, a smartphone, or a tablet computer. The terminal 42 includes a display 501, a user I / F 502, a CPU 503, a RAM 504, a network I / F 505, a flash memory 506, an audio I / O (Input / Output) 507, and a microphone 508.

[0085] The CPU 503 is a control unit that controls the operation of the terminal 42. The CPU 503 performs various operations by reading out a predetermined program stored in a flash memory 506, which is a storage medium, into the RAM 504 and executing it.

[0086] The programs read by the CPU 503 do not need to be stored in the flash memory 506 of the device itself. For example, the programs may be stored in a storage medium of an external device such as a server. In this case, the CPU 503 only needs to read the programs from the server into the RAM 504 and execute them.

[0087] The CPU 503 performs signal processing on the audio signal input via the network I / F 505. The CPU 503 outputs the processed audio signal to the headphones 43 via the audio I / O 507.

[0088] The CPU 503 receives distribution data from the distribution device 12 via the network I / F 505. The CPU 503 renders the distribution data and provides the sounds of the performers and sounds related to the reverberation of the space to the listeners in the fifth venue 20B.

[0089] Specifically, CPU 503 convolves a head-related transfer function (hereinafter referred to as HRTF) into the sound signal related to the sound of the performer, and performs sound image localization processing (binaural processing) so that the sound of the performer is localized at the position of the performer. HRTF corresponds to the transfer function between a predetermined position and the ears of the listener. HRTF is a transfer function that expresses the volume, arrival time, and frequency characteristics of a sound from a sound source at a certain position to the left and right ears, respectively. CPU 503 convolves the HRTF into the sound signal of the performer's sound based on the position of the performer. As a result, the sound of the performer is localized at a position according to the position information.

[0090] Furthermore, the CPU 503 performs indirect sound generation processing by binaural processing that convolves the sound signal of the performer's sound with HRTF corresponding to the reverberation information of the space. The CPU 503 localizes the early reflection sound and the rear reverberation sound by convolving the HRTF from the position of the virtual sound source corresponding to each early reflection sound included in the reverberation information of the space to the left and right ears. However, the rear reverberation sound is a reflection sound whose arrival direction is not fixed. Therefore, the CPU 503 may perform effect processing such as reverb without performing localization processing on the rear reverberation sound. The CPU 503 may perform digital filter processing (headphone inverse characteristic processing) that reproduces the inverse characteristics of the acoustic characteristics of the headphones 43 used by the listener.

[0091] Furthermore, CPU 503 renders ambience information from the distribution data to provide the environmental sounds generated in first venue 10 to the listener in fifth venue 20B. If the ambience information includes position information of the environmental sounds, CPU 503 performs localization processing using HRTF, and performs effect processing on sounds whose arrival direction is uncertain.

[0092] Furthermore, the CPU 503 may render the video signal of the distribution data and display live video on the display 501.

[0093] This allows the terminal 42 to provide the listeners in the fifth venue 20B with the same sense of realism as the first venue 10.

[0094] The terminal 42 also acquires the sound of the listener in the fifth venue 20B via the microphone 508. The terminal 42 transmits the sound of the listener to another device. For example, the terminal 42 feeds back the sound of the listener to the first venue 10. Alternatively, the terminal 42 may display icon images such as "cheering," "applause," "calling," and "bustling noise" on the display 501 and receive a selection operation on these icon images from the listener via the user I / F 502 to receive a reaction. The terminal 42 generates a sound corresponding to the received reaction and transmits the generated sound as ambience information to another device. Alternatively, the terminal 42 may transmit information indicating the type of environmental sound such as the listener's cheering, applause, or call as ambience information. In this case, the receiving device (for example, the distribution device 12 and the mixer 11) generates a corresponding sound signal based on the ambience information and provides the listener's cheering, applause, or call sound in the venue.

[0095] Terminal 42 may also receive the position information of the listener via user I / F 502. Terminal 42 transmits the received position information of the listener to first venue 10. Based on the sound and position information of the listener in third venue 20A received from AV receiver 32, distribution device 12 and mixer 11 in the first venue perform processing to localize the sound of the listener to a specified position.

[0096] The terminal 42 may also change the HRTF based on position information received from the user. For example, if the listener specifies a position immediately in front of the stage in the first venue 10, the terminal 42 sets the position of the performer's sound to a position immediately in front of the listener, and convolutes an HRTF that localizes the performer's sound to that position. This allows the listener in the fifth venue 20B to experience the same sense of presence as if they were immediately in front of the stage in the first venue 10.

[0097] The sound of the listener in the fifth venue 20B may be transmitted to the second venue 20, not to the first venue 10, or may be transmitted to other venues. As described above, the sound of the listener in the fifth venue 20B may be transmitted only to the friend's house (the fourth venue). This allows the listener in the fifth venue 20B and the listener in the fourth venue to watch the live performance in the first venue 10 while talking to each other.

[0098] Furthermore, in the live data distribution system of this embodiment, multiple users can specify the same position. For example, multiple users may each specify a position immediately in front of the stage in the first venue 10. In this case, each listener can experience a sense of realism as if they were in a position immediately in front of the stage. This allows multiple listeners to watch a performer's performance with the same sense of realism for one position (seat in the venue). In this case, the live event organizer can provide services that exceed the capacity of the actual space to accommodate more spectators.

[0099] Fig. 15 is a block diagram showing the configuration of a live data distribution system 1D according to Modification 4. Components common to Fig. 1 are given the same reference numerals and description thereof will be omitted.

[0100] The live data distribution system 1D further includes a server 50 and a terminal 55. The terminal 55 is installed in the sixth venue 10A. The server 50 is an example of a distribution device, and the hardware configuration of the server 50 is similar to that of the distribution device 12. The hardware configuration of the terminal 55 is similar to that of the terminal 42 shown in FIG. 14.

[0101] The sixth venue 10A is a home or the like of a performer who performs a performance such as a musical instrument remotely. The performer in the sixth venue 10A performs a performance such as a musical instrument or a singing in accordance with the musical instrument or the singing in the first venue. The terminal 55 transmits the sound of the performer in the sixth venue 10A to the server 50. The terminal 55 may also capture an image of the performer in the sixth venue 10A with a camera (not shown) and transmit the video signal to the server 50.

[0102] The server 50 distributes distribution data including the sounds of the performers in the first venue 10, the sounds of the performers in the sixth venue 10A, reverberation information of the space in the first venue 10, ambience information of the first venue 10, live footage of the first venue 10, and footage of the performers in the sixth venue 10A.

[0103] In this case, the playback device 22 renders the distribution data and provides the sounds of the performers in the first venue 10, the sounds of the performers in the sixth venue 10A, the reverberation of the space in the first venue 10, the ambient sounds in the first venue 10, the live video of the first venue 10, and the video of the performers in the sixth venue 10A to the second venue 20. For example, the playback device 22 displays the video of the performers in the sixth venue 10A superimposed on the live video of the first venue 10.

[0104] The sound of the performer in the sixth venue 10A does not need to be localized, but may be localized to a position that matches the video displayed on the display. For example, if the performer in the sixth venue 10A is displayed on the right side of the live video, the sound of the performer in the sixth venue 10A is localized to the right.

[0105] Alternatively, the performer in the sixth venue 10A or the distributor of the distribution data may specify the position of the performer. In this case, the distribution data includes position information of the performer in the sixth venue 10A. The playback device 22 localizes the sound of the performer in the sixth venue 10A based on the position information of the performer in the sixth venue 10A.

[0106] The video of the performers in the sixth venue 10A is not limited to video captured by a camera. For example, a character image (virtual video) made of a two-dimensional image or 3D modeling may be distributed as the video of the performers in the sixth venue 10A.

[0107] The distribution data may include sound recording data. The distribution data may also include video recording data. For example, the distribution device may distribute distribution data including the sound of the performer at the first venue 10, the sound recording data, the reverberation information of the space of the first venue 10, the ambience information of the first venue 10, the live video of the first venue 10, and the video recording data. In this case, the playback device renders the distribution data and provides the sound of the performer at the first venue 10, the sound related to the sound recording data, the reverberation of the space of the first venue 10, the environmental sound of the first venue 10, the live video of the first venue 10, and the video related to the video recording data to other venues. The playback device 22 displays the video of the performer corresponding to the video recording data by superimposing it on the live video of the first venue 10.

[0108] The distribution device may also determine the type of instrument when recording the sound related to the sound recording data. In this case, the distribution device distributes the sound recording data including information indicating the type of instrument determined to be the sound recording data. The playback device generates an image of the corresponding instrument based on the information indicating the type of instrument. The playback device may display the image of the instrument superimposed on the live video of the first venue 10.

[0109] Furthermore, the distribution data does not need to superimpose the video of the performer in the sixth venue 10A on the live video of the first venue 10. For example, the distribution data may distribute the video of each performer in the first venue 10 and the sixth venue 10A and the background video as separate data. In this case, the distribution data includes information indicating the display position of each video. The playback device renders the video of each performer based on the information indicating the display position.

[0110] Furthermore, the background image is not limited to an image of a venue where a live performance is actually taking place, such as first venue 10. The background image may be an image of a venue other than the venue where the live performance is taking place.

[0111] Furthermore, the spatial reverberation information included in the distribution data does not need to correspond to the spatial reverberation of the first venue 10. For example, the spatial reverberation information may be virtual space information for virtually reproducing the spatial reverberation of the venue corresponding to the background video (information indicating the size, shape, and wall material of each venue, or an impulse response indicating the transfer function of each venue). The impulse response of each venue may be measured in advance, or may be obtained by simulation from the size, shape, and wall material of each venue.

[0112] Furthermore, the ambience information may also be changed to contents corresponding to the background video. For example, in the case of a background video of a large venue, the ambience information includes sounds such as cheers, applause, and cheers from many listeners. Furthermore, an outdoor venue includes background noise different from that of an indoor venue. Furthermore, the reverberation of the environmental sounds may also change according to the reverberation information of the space. Furthermore, the ambience information may include information indicating the number of spectators and information indicating the degree of congestion (the density of people). The playback device increases or decreases the number of sounds such as cheers, applause, and cheers from listeners based on the information indicating the number of spectators. Furthermore, the playback device increases or decreases the volume of the cheers, applause, and cheers from listeners based on the information indicating the degree of congestion.

[0113] Alternatively, the ambience information may be changed depending on the performer. For example, when a performer with many female fans gives a live performance, the sounds of the listeners' cheers, calls, cheers, etc. included in the ambience information are changed to female voices. The ambience information may contain sound signals of the listeners' voices, but may also contain information indicating the attributes of the audience, such as the male / female ratio or age ratio. The playback device changes the voice quality of the listeners' cheers, applause, cheers, etc. based on the information indicating the attributes.

[0114] Furthermore, the listeners at each venue may specify background video and spatial reverb information by using the user I / F of the playback device.

[0115] FIG. 16 is a diagram showing an example of live video 700 displayed on a playback device in each venue. The live video 700 is composed of video captured at the first venue 10 or other venues, or virtual video (computer graphics) corresponding to each venue. The live video 700 is displayed on a display of the playback device. The live video 700 displays the background of the venue, the stage, performers including musical instruments, and video of listeners in the venue. The video of the background of the venue, the stage, performers including musical instruments, and listeners in the venue may all be actually captured video or may be virtual video. Moreover, only the background video may be actually captured video, and the other videos may be virtual video. Moreover, the live video 700 displays an icon image 751 and an icon image 752 for designating a space. The icon image 751 is an image for designating the space of a certain venue, Stage A (for example, the first venue 10), and the icon image 752 is an image for designating the space of another venue, Stage B (for example, another concert hall, etc.). Furthermore, in the live video 700, a listener image 753 for specifying the listener's position is displayed.

[0116] A listener using the playback device specifies a desired space by specifying either icon image 751 or icon image 752 using the user I / F of the playback device. The distribution device distributes the background video and spatial reverb information corresponding to the specified space by including them in the distribution data. Alternatively, the distribution device may distribute a plurality of background videos and spatial reverb information by including them in the distribution data. In this case, the playback device renders the background video and spatial reverb information corresponding to the space specified by the listener from the received distribution data.

[0117] In the example of Fig. 16, icon image 751 is specified. The playback device displays a background video (e.g., a video of first venue 10) corresponding to Stage A of icon image 751, and plays back sounds related to the reverberation of the space corresponding to the specified Stage A. When the listener specifies icon image 752, the playback device switches to and displays a background video of Stage B, which is another space corresponding to icon image 752, and plays back sounds related to the reverberation of the corresponding other space based on the virtual space information corresponding to Stage B.

[0118] This allows the listener of each playback device to experience the same sense of realism as if they were watching a live performance in a desired space.

[0119] Furthermore, the listener of each playback device can specify a desired position within the venue by moving the listener image 753 in the live video 700. The playback device performs localization processing based on the position specified by the user. For example, if the listener moves the listener image 753 to a position just in front of the stage, the playback device sets the localization position of the performer's sound to a position just in front of the listener, and performs localization processing so that the performer's sound is localized at that position. This allows the listener of each playback device to feel as if they are right in front of the stage.

[0120] As described above, when the position of the sound source and the position of the listener (position of the sound receiving point) change, the sound related to the reverberation of the space also changes. The playback device can obtain the early reflected sound by calculation even when the space changes, the position of the sound source changes, or the position of the sound receiving point changes. Therefore, even if the impulse response or the like is not measured in the actual space, the playback device can obtain the sound related to the reverberation of the space based on the virtual space information. Therefore, the playback device can realize the reverberation occurring in spaces including real spaces with high accuracy.

[0121] For example, mixer 11 may function as a distribution device, and mixer 21 may function as a playback device. Also, playback devices do not need to be installed at each venue. For example, server 50 shown in Fig. 15 may render the distribution data and distribute the processed audio signals to terminals at each venue. In this case, server 50 functions as a playback device.

[0122] The sound source information may include information indicating the posture of the performer (e.g., the left / right direction of the performer). The playback device may adjust the volume or frequency characteristics based on the posture information of the performer. For example, the playback device may perform a process that reduces the volume as the performer faces further left / right, based on the case where the performer faces directly ahead. The playback device may also perform a process that attenuates high frequencies more than low frequencies as the performer faces further left / right. This allows the sound to change according to the performer's posture, allowing listeners to enjoy a more realistic live performance.

[0123] Next, Fig. 17 is a block diagram showing an application example of signal processing performed by a playback device. In this example, rendering is performed using the terminal 42 and headphones 43 shown in Fig. 13. The playback device (terminal 42 in the example of Fig. 13) functionally comprises an instrument model processing unit 551, an amplifier model processing unit 552, a speaker model processing unit 553, a space model processing unit 554, a binaural processing unit 555, and a headphone inverse characteristic processing unit 556.

[0124] The instrument model processing unit 551, the amplifier model processing unit 552, and the speaker model processing unit 553 perform signal processing to impart the acoustic characteristics of the audio equipment to the sound signal related to the performance sound. A first digital signal processing model for performing the signal processing is included in, for example, the sound source information distributed by the distribution device 12. The first digital signal processing model is a digital filter that simulates the acoustic characteristics of the instrument, the acoustic characteristics of the amplifier, and the acoustic characteristics of the speaker, respectively. The first digital signal processing model is created in advance by a manufacturer of the instrument, a manufacturer of the amplifier, and a manufacturer of the speaker through simulation or the like. The instrument model processing unit 551, the amplifier model processing unit 552, and the speaker model processing unit 553 perform digital filter processing that simulates the acoustic characteristics of the instrument, the acoustic characteristics of the amplifier, and the acoustic characteristics of the speaker, respectively. Note that, when the instrument is an electronic instrument such as a synthesizer, the instrument model processing unit 551 inputs note event data (information indicating the timing of sound generation, pitch, etc. of the sound to be generated) instead of a sound signal, and generates a sound signal having the acoustic characteristics of an electronic instrument such as a synthesizer.

[0125] This allows the playback device to reproduce the acoustic characteristics of any musical instrument. For example, in FIG. 16, a live video 700 of a virtual image (computer graphics) is displayed. Here, a listener using the playback device may change the image of another virtual musical instrument using the user I / F of the playback device. When the listener changes the instrument displayed in the live video 700 to the image of another musical instrument, the musical instrument model processing unit 551 of the playback device performs signal processing according to the first digital signal processing model corresponding to the changed instrument. This allows the playback device to output a sound that reproduces the acoustic characteristics of the musical instrument displayed in the live video 700.

[0126] Similarly, a listener using the playback device may change the type of amplifier and the type of speaker to different types using the user I / F of the playback device. The amplifier model processing unit 552 and the speaker model processing unit 553 perform digital filter processing that simulates the acoustic characteristics of the changed type of amplifier and the acoustic characteristics of the speaker. The speaker model processing unit 553 may simulate the acoustic characteristics for each speaker direction. In this case, a listener using the playback device may change the orientation of the speaker using the user I / F of the playback device. The speaker model processing unit 553 performs digital filter processing according to the changed speaker orientation.

[0127] The space model processor 554 is a second digital signal processing model that reproduces the acoustic characteristics of a room in a live venue (for example, the above-mentioned spatial reverberation). The second digital signal processing model may be obtained, for example, by using a test sound in an actual live venue. Alternatively, the second digital signal processing model may obtain the delay amount and level of a virtual sound source by calculation from virtual space information (information indicating the size, shape, and material of the walls of each venue) as described above.

[0128] When the position of the sound source and the position of the listener (position of the sound receiving point) change, the sound related to the reverberation of the space also changes. The playback device can calculate the delay amount and level of the imaginary sound source even when the space changes, the position of the sound source changes, or the position of the sound receiving point changes. Therefore, even if the impulse response or the like is not measured in the actual space, the playback device can calculate the sound related to the reverberation of the space based on the virtual space information. Therefore, the playback device can realize the reverberation occurring in spaces including real spaces with high accuracy.

[0129] The virtual space information may include information on the position and material of structures (acoustic obstacles) such as pillars. In the sound source localization and indirect sound generation process, if an obstacle exists in the path of the direct sound and indirect sound arriving from the sound source, the playback device reproduces the phenomena of reflection, obstruction, and diffraction caused by the obstacle.

[0130] FIG. 18 is a schematic diagram showing the path of a sound that arrives at a sound receiving point 75 after being reflected from a wall surface from a sound source 70. The sound source 70 shown in FIG. 18 may be either a performance sound (first sound source) or an environmental sound (second sound source). The playback device determines the position of an imaginary sound source 70A that exists with the wall surface as a mirror surface with respect to the position of the sound source 70 based on the position of the sound source 70, the position of the wall surface, and the position of the sound receiving point 75. Then, the playback device determines the delay amount of the imaginary sound source 70A based on the distance from the imaginary sound source 70A to the sound receiving point 75. In addition, the playback device determines the level of the imaginary sound source 70A based on information on the material of the wall surface. Furthermore, when an obstacle 77 exists on the path from the position of the imaginary sound source 70A to the sound receiving point 75 as shown in FIG. 18, the playback device determines the frequency characteristics caused by diffraction of the obstacle 77. For example, high-frequency sounds are attenuated by diffraction. 18, when an obstacle 77 is present on the path from the position of the imaginary sound source 70A to the sound receiving point 75, the playback device performs equalizer processing to reduce the high frequency level. The frequency characteristics caused by diffraction may be included in the virtual space information.

[0131] The reproducing device may set new second imaginary sound source 77A and third imaginary sound source 77B at the left and right positions of the obstacle 77. The second imaginary sound source 77A and the third imaginary sound source 77B correspond to new sound sources generated by diffraction. The second imaginary sound source 77A and the third imaginary sound source 77B are sounds to which frequency characteristics generated by diffraction are added to the sound of the imaginary sound source 70A. The reproducing device recalculates the delay amount and the level based on the positions of the second imaginary sound source 77A and the third imaginary sound source 77B and the position of the sound receiving point 75. This makes it possible to reproduce the diffraction phenomenon of the obstacle 77.

[0132] The playback device may calculate the delay and level of the sound from the imaginary sound source 70A that is reflected by the obstacle 77, then reflected by a wall surface, and reaches the sound receiving point 75. Furthermore, when the playback device determines that the imaginary sound source 70A is blocked by the obstacle 77, it may erase the imaginary sound source 70A. Information for determining whether or not to block the imaginary sound source 70A may be included in the virtual space information.

[0133] By performing the above processing, the playback device performs first digital signal processing that represents the acoustic characteristics of the audio equipment and second digital signal processing that represents the acoustic characteristics of the room, thereby generating sound related to the sound of the sound source and the reverberation of the space.

[0134] The binaural processing unit 555 convolves a head-related transfer function (hereinafter referred to as HRTF) with the sound signal to perform sound image localization processing of the sound source and various indirect sounds. The headphone inverse characteristic processing unit 556 performs digital filtering to reproduce the inverse characteristics of the acoustic characteristics of the headphones used by the listener.

[0135] Through the above processing, the user can obtain a sense of realism as if he or she were watching a live performance in a desired space with desired audio equipment.

[0136] The playback device does not need to include all of the musical instrument model processing unit 551, the amplifier model processing unit 552, the speaker model processing unit 553, and the space model processing unit 554 shown in FIG. 17. The playback device may execute signal processing using at least one digital signal processing model. The playback device may execute signal processing using one digital signal processing model for one sound signal (e.g., the sound of a certain performer), or may execute signal processing using one digital signal processing model for each of multiple sound signals. The playback device may execute signal processing using multiple digital signal processing models for one sound signal (e.g., the sound of a certain performer), or may execute signal processing using multiple digital signal processing models for multiple sound signals. The playback device may execute signal processing using a digital signal processing model for environmental sound.

[0137] The description of the present embodiment is illustrative in all respects and is not restrictive. The scope of the present invention is indicated by the claims, not by the above-described embodiments. Furthermore, the scope of the present invention is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0138] 1, 1A, 1B, 1C, 1D...Live data distribution system 5. Internet 10…1st Venue 10A…6th Venue 11…Mixer 12...Distribution device 13A~13F…Microphone 14A~14G…Speakers 15A~15C…Tracker 16…Camera 20…Second Venue 20A…Venue 3 20B…5th venue 21…Mixer 22...Playback device 23...Indicator 24A~24F…Speakers 25A~25C…Microphone 32...AV receiver 33...Indicator 34A…Speaker 35…Mike 42...Terminal 43...Headphones 50…Server 55…Terminal 101...Indicator 102...User I / F 103…Audio I / O 104...signal processing unit 105…Network I / F 106…CPU 107...Flash memory 108…RAM 201…Display unit 202...User I / F 203...CPU 204…RAM 205…Network I / F 206...Flash memory 207…General-purpose communication I / F 301...Indicator 302...User I / F 303...CPU 304...RAM 305…Network I / F 306...Flash memory 307...Video I / F 401...Indicator 402...User I / F 403...Audio I / O 404...signal processing unit 405...Network I / F 406…CPU 407...Flash memory 408...RAM 409…Video I / F 501...Indicator 503…CPU 504…RAM 505…Network I / F 506...Flash memory 507…Audio I / O 508…Mike 700…Live footage

Claims

1. Distributing, as distribution data, sound source information relating to sounds generated in the first venue and spatial reverberation information, which is information relating to early reflection sounds that change depending on the position of the sound; Rendering the distribution data to provide a sound related to the sound source information and a sound related to the reverberation of the space to a second venue. A live data distribution method, comprising: the spatial reverberation information includes information for generating indirect sounds, The rendering includes a process of generating indirect sound of the sound source, The process of generating the indirect sound includes a process of generating an early reflection sound, The generated early reflection sound changes depending on the position of the sound, The spatial reverberation information further includes information about a rear reverberation sound that does not change depending on the position of the sound, The process of generating the indirect sound further includes a process of generating the rear reverberation sound. Live data delivery methods.

2. The sound source information includes a sound signal of a sound generated in the first venue and position information of the sound, The rendering includes a localization process according to a position of the sound. The live data distribution method according to claim 1 .

3. The spatial reverberation information includes an impulse response that changes depending on the position of the sound.

3. The live data distribution method according to claim 1 or 2.

4. Ambience information relating to environmental sounds is included in the distribution data and distributed; The rendering further includes providing the ambient sound. The live data distribution method according to any one of claims 1 to 3.

5. The ambience information includes position information of the environmental sound, In the rendering, a localization process is performed to localize the environmental sound to a position of the position information based on the ambience information, and an effect process is performed to cause a user to perceive a spatial spread other than the localization process. The live data distribution method according to claim 4.

6. The spatial reverberation information includes virtual space information for reproducing a reverberation other than the reverberation of the first venue, The rendering reproduces sounds related to the reverberation of the space based on the virtual space information. The live data distribution method according to any one of claims 1 to 5.

7. Accepting an operation to designate a space from a user of the second venue; The rendering reproduces a sound related to the reverberation of the space based on the virtual space information corresponding to the space received by the operation. The live data distribution method according to claim 6.

8. Providing live footage to the second venue; The virtual space information corresponds to the spatial reverberation of the live video.

8. The live data distribution method according to claim 6 or 7.

9. Rendering the distribution data to provide a sound related to the sound source information and a sound related to the reverberation of the space to a third venue; The sound related to the reverberation of the space is common to the second venue and the third venue, The live data distribution method according to any one of claims 1 to 8.

10. the sound source information includes a first digital signal processing model that represents an acoustic characteristic of an acoustic device; providing a sound related to the sound source information to the second venue using the first digital signal processing model; The live data distribution method according to any one of claims 1 to 9.

11. the spatial reverberation information includes a second digital signal processing model representing an acoustic characteristic of a room; providing a sound related to the reverberation of the space to the second venue using the second digital signal processing model; The live data distribution method according to any one of claims 1 to 10.

12. Distributing location information indicating the location of a structure installed in the first venue; The sound source information includes a sound signal of a sound generated in the first venue and position information of the sound, The rendering includes signal processing based on position information indicating a position of the structure and position information of the sound. The live data distribution method according to any one of claims 1 to 11.

13. a live data distribution device that distributes sound source information related to sounds generated in the first venue and spatial reverberation information as distribution data; a live data reproducing device that renders the distribution data and provides a sound related to the sound source information and a sound related to the reverberation of the space to a second venue; A live data distribution system comprising: the spatial reverberation information includes information for generating indirect sounds, The rendering includes a process of generating an indirect sound of the sound source. The process of generating the indirect sound includes a process of generating an early reflection sound, The generated early reflection sound changes depending on the position of the sound, The spatial reverberation information further includes information about a rear reverberation sound that does not change depending on the position of the sound, The process of generating the indirect sound further includes a process of generating the rear reverberation sound. Live data distribution system.

14. The sound source information includes a sound signal of a sound generated in the first venue and position information of the sound, The rendering includes a localization process according to a position of the sound. The live data distribution system according to claim 13.

15. The spatial reverberation information includes an impulse response that changes depending on the position of the sound. The live data distribution system according to claim 13 or 14.

16. the live data distribution device distributes ambience information related to environmental sounds by including it in the distribution data; The rendering further includes providing the ambient sound. The live data distribution system according to any one of claims 13 to 15.

17. The ambience information includes position information of the environmental sound, In the rendering, a localization process is performed to localize the environmental sound to a position of the position information based on the ambience information, and an effect process is performed to cause a user to perceive a spatial spread other than the localization process. The live data distribution system according to claim 16.

18. The spatial reverberation information includes virtual space information for reproducing a reverberation other than the reverberation of the first venue, The rendering reproduces sounds related to the reverberation of the space based on the virtual space information. The live data distribution system according to any one of claims 13 to 17.

19. the live data reproducing device accepts an operation for designating a space from a user of the second venue; The rendering reproduces a sound related to the reverberation of the space based on the virtual space information corresponding to the space received by the operation. The live data distribution system according to claim 18.

20. the live data reproducing device provides live video to the second venue; The virtual space information corresponds to the spatial reverberation of the live video.

20. The live data distribution system according to claim 18 or 19.

21. the live data reproducing device renders the distribution data to provide a sound related to the sound source information and a sound related to the reverberation of the space to a third venue; The sound related to the reverberation of the space is common to the second venue and the third venue, The live data distribution system according to any one of claims 13 to 20.

22. the sound source information includes a first digital signal processing model that represents an acoustic characteristic of an acoustic device; the live data reproducing device provides the sound related to the sound source information to the second venue using the first digital signal processing model; The live data distribution system according to any one of claims 13 to 21.

23. the spatial reverberation information includes a second digital signal processing model representing an acoustic characteristic of a room; the live data reproduction device provides the sound related to the reverberation of the space to the second venue using the second digital signal processing model; The live data distribution system according to any one of claims 13 to 22.

24. the live data distribution device distributes location information indicating locations of structures installed in the first venue; The sound source information includes a sound signal of a sound generated in the first venue and position information of the sound, The rendering includes signal processing based on position information indicating a position of the structure and position information of the sound. The live data distribution system according to any one of claims 13 to 23.

25. The sound source information related to the sound generated in the first venue and the spatial reverberation information are distributed as distribution data. causing a playback device to render the distribution data and provide the sound related to the sound source information and the sound related to the reverberation of the space to the second venue; A live data distribution device, comprising: the spatial reverberation information includes information for generating indirect sounds, The rendering includes a process of generating an indirect sound of the sound source. The process of generating the indirect sound includes a process of generating an early reflection sound, The generated early reflection sound changes depending on the position of the sound, The spatial reverberation information further includes information about a rear reverberation sound that does not change depending on the position of the sound, The process of generating the indirect sound further includes a process of generating the rear reverberation sound. Live data distribution device.

26. receiving, from a live data distribution device that distributes sound source information and spatial reverberation information related to sounds generated in the first venue as distribution data, the distribution data; Rendering the distribution data to provide a sound related to the sound source information and a sound related to the reverberation of the space to a second venue. A live data playback device, comprising: the spatial reverberation information includes information for generating indirect sounds, The rendering includes a process of generating an indirect sound of the sound source. The process of generating the indirect sound includes a process of generating an early reflection sound, The generated early reflection sound changes depending on the position of the sound, The spatial reverberation information further includes information about a rear reverberation sound that does not change depending on the position of the sound, The process of generating the indirect sound further includes a process of generating the rear reverberation sound. Live data playback device.

27. The sound source information related to the sound generated in the first venue and the spatial reverberation information are distributed as distribution data. causing a playback device to render the distribution data and provide the sound related to the sound source information and the sound related to the reverberation of the space to the second venue; A live data distribution method, comprising: the spatial reverberation information includes information for generating indirect sounds, The rendering includes a process of generating an indirect sound of the sound source. The process of generating the indirect sound includes a process of generating an early reflection sound, The generated early reflection sound changes depending on the position of the sound, The spatial reverberation information further includes information about a rear reverberation sound that does not change depending on the position of the sound, The process of generating the indirect sound further includes a process of generating the rear reverberation sound. Live data delivery methods.

28. receiving, from a live data distribution device that distributes sound source information and spatial reverberation information related to sounds generated in the first venue as distribution data, the distribution data; Rendering the distribution data to provide a sound related to the sound source information and a sound related to the reverberation of the space to a second venue. A live data playback method, comprising: the spatial reverberation information includes information for generating indirect sounds, The rendering includes a process of generating an indirect sound of the sound source. The process of generating the indirect sound includes a process of generating an early reflection sound, The generated early reflection sound changes depending on the position of the sound, The spatial reverberation information further includes information about a rear reverberation sound that does not change depending on the position of the sound, The process of generating the indirect sound further includes a process of generating the rear reverberation sound. Live data playback method.

Images