Field data transmission method, field data transmission system, transmission device thereof, field data playback device and playback method thereof

Abstract

The on-site data transmission method transmits first sound source information related to the sound of a first sound source occurring at a first location of a first venue and the position information of the first sound source, and second sound source information related to the second sound source occurring at a second location of the first venue as transmission data, reproduces the transmission data, and provides the sound of the first sound source that has been subjected to positioning processing based on the position information of the first sound source and the sound of the second sound source to a second venue.

Description

Technical Field

[0001] One embodiment of the present invention relates to a live data transmission method, a live data transmission system, a live data transmission device, a live data playback device, and a live data playback method. Background Art

[0002] Patent Document 1 discloses a method for watching a sports game in a terminal for watching the game, which allows a user to effectively enjoy the excitement of the game as if he or she were actually in a stadium.

[0003] The game watching method of Patent Document 1 is to transmit reaction information indicating the reaction of each user from the terminal of each user, and each user's terminal displays icon information based on the reaction information.

[0004] Patent Document 1: Japanese Patent Application Publication No. 2019-024157 Summary of the invention

[0005] The system of Patent Document 1 only displays icon information, and is not a system that provides a sense of presence of the live venue to the venue of the transmission destination when transmitting the live data.

[0006] An object of one embodiment of the present invention is to provide a live data transmission method, a live data transmission system, a live data transmission device, a live data playback device and a live data playback method, which can provide the sense of presence of the live venue to the venue of the transmission target when transmitting live data.

[0007] The on-site data transmission method transmits first sound source information related to the sound of a first sound source occurring at a first location of a first venue and the position information of the first sound source, and second sound source information related to a second sound source occurring at a second location of the first venue as transmission data, reproduces the transmission data, and provides the sound of the first sound source that has been subjected to positioning processing based on the position information of the first sound source and the sound of the second sound source to a second venue.

[0008] Effects of the Invention

[0009] The on-site data transmission method can provide the sense of presence of the on-site venue to the venue of the transmission destination when transmitting the on-site data. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a block diagram showing the structure of a live data transmission system 1.

[0011] Figure 2 This is a schematic diagram of the floor plan of the first venue 10.

[0012] Figure 3 This is a schematic diagram of the floor plan of the second venue 20.

[0013] Figure 4 : is a block diagram showing the structure of the mixer 11.

[0014] Figure 5 It is a block diagram showing the structure of the transmission device 12.

[0015] Figure 6 It is a flowchart showing the operation of the conveying device 12.

[0016] Figure 7 It is a block diagram showing the structure of the playback device 22.

[0017] Figure 8 It is a flowchart showing the operation of the playback device 22.

[0018] Fig. 9 This is a block diagram showing the configuration of a field data transmission system 1A according to Modification 1.

[0019] Fig.10 1A is a schematic plan view of a second venue 20 of an on-site data transmission system 1A according to Modification 1. FIG.

[0020] Fig.11 This is a block diagram showing the configuration of a field data transmission system 1B according to Modification 2.

[0021] Fig.12 It is a block diagram showing the structure of the AV receiver 32.

[0022] Fig.13 It is a block diagram showing the structure of a field data transmission system 1C according to Modification 3.

[0023] Fig.14 It is a block diagram showing the structure of the terminal 42.

[0024] Fig.15 It is a block diagram showing the structure of a field data transmission system 1D according to Modification 4.

[0025] Fig.16 This is a diagram showing an example of a live video 700 displayed on a playback device at each venue.

[0026] Fig.17 This is a block diagram showing an application example of signal processing performed by a playback device.

[0027] Fig.18 Schematic diagram showing the path of sound that is reflected from the wall surface from the sound source 70 and reaches the sound collecting point 75 . DETAILED DESCRIPTION

[0028] Figure 1 2 is a block diagram showing the configuration of the on-site data transmission system 1. The on-site data transmission system 1 is composed of a plurality of audio equipment and information processing devices installed in the first venue 10 and the second venue 20, respectively.

[0029] Figure 2 This is a floor plan of the first venue 10. Figure 3 1 is a schematic plan view of the second venue 20. In this example, the first venue 10 is a live venue where performers perform (performances), and the second venue 20 is a public viewing venue where remote audiences watch and listen to the performers' performances.

[0030] The first venue 10 is provided with a mixer 11, a transmission device 12, and a plurality of microphones 13A to 13B.

[0031] 13F, a plurality of speakers 14A to 14G, a plurality of trackers 15A to 15C, and a camera 16. A mixer 21, a playback device 22, a display 23, and a plurality of speakers 24A to 24F are provided at the second venue 20. The transmission device 12 and the playback device 22 are connected via the Internet 5. In addition, the number of microphones, the number of speakers, and the number of trackers are not limited to the numbers shown in this embodiment. In addition, the arrangement of microphones and speakers is not limited to the examples shown in this embodiment.

[0032] The mixer 11 is connected to the transmission device 12, a plurality of microphones 13A to 13F, a plurality of speakers 14A to 14G, and a plurality of trackers 15A to 15C. The mixer 11, the plurality of microphones 13A to 13F, and the plurality of speakers 14A to 14G are connected via a network cable or an audio cable. The plurality of trackers 15A to 15C are connected to the mixer 11 via wireless communication. The mixer 11 and the transmission device 12 are connected via a network cable. In addition, the transmission device 12 is connected to the camera 16 via an image cable. The camera 16 captures live images including performers.

[0033] A plurality of speakers 14A to 14G are arranged along the wall surface of the first venue 10. The first venue 10 of this example is rectangular in shape when viewed from above. A stage is arranged in front of the first venue 10. On the stage, performers perform such as singing or playing music. The speaker 14A is arranged on the left side of the stage, the speaker 14B is arranged in the center of the stage, and the speaker 14C is arranged on the right side of the stage. The speaker 14D is arranged on the left side of the front and back center of the first venue 10, and the speaker 14E is arranged on the right side of the front and back center of the first venue 10. The speaker 14F is arranged on the left side of the rear of the first venue 10, and the speaker 14G is arranged on the right side of the rear of the first venue 10.

[0034] Microphone 13A is installed on the left side of the stage, microphone 13B is installed in the center of the stage, and microphone 13C is installed on the right side of the stage. Microphone 13D is installed on the left side of the front and back center of the first venue 10, and microphone 13E is installed in the rear center of the first venue 10. Microphone 13F is installed on the right side of the front and back center of the first venue 10.

[0035] The mixer 11 receives sound signals from the microphones 13A to 13F. In addition, the mixer 11 outputs sound signals to the speakers 14A to 14G. In this embodiment, a speaker and a microphone are shown as an example of an audio device connected to the mixer 11, but in fact, a plurality of audio devices are connected to the mixer 11. The mixer 11 receives sound signals from a plurality of audio devices such as microphones, performs signal processing such as mixing, and outputs sound signals to a plurality of audio devices such as speakers.

[0036] The microphones 13A to 13F acquire the singing sound or the playing sound of each performer as the sound generated in the first venue 10. Alternatively, the microphones 13A to 13F acquire the ambient sound of the first venue 10. Figure 2 In the example, microphones 13A to 13C acquire the performer's voice, and microphones 13D to 13F acquire the ambient sound. The ambient sound includes the audience's support, applause, shouting, cheering, chorus, or noise. However, the performer's voice can be input by line. Line input means that the sound output from a sound source such as a musical instrument is not input by the microphone picking up the sound, but the sound signal is input from an audio cable connected to the sound source. The performer's voice is preferably acquired with a high SN ratio and does not include other sounds.

[0037] The speakers 14A to 14G output the performer's voice to the first venue 10. The speakers 14A to 14G may also output the initial reflected sound or the rear reverberation sound for controlling the sound field of the first venue 10.

[0038] The mixer 21 of the second venue 20 is connected to a playback device 22 and a plurality of speakers 24A to 24F. These audio devices are connected via a network cable or an audio cable. In addition, the playback device 22 is connected to a display 23 via a video cable.

[0039] A plurality of speakers 24A to 24F are arranged along the wall surface of the second venue 20. The second venue 20 of this example is rectangular in shape when viewed from above. A display 23 is arranged in front of the second venue 20. The display 23 displays the live image captured in the first venue 10. The speaker 24A is arranged on the left side of the display 23, and the speaker 24B is arranged on the right side of the display 23. The speaker 24C is arranged on the left side of the front and back center of the second venue 20, and the speaker 24D is arranged on the right side of the front and back center of the second venue 20. The speaker 24E is arranged on the left side of the rear of the second venue 20, and the speaker 24F is arranged on the right side of the rear of the second venue 20.

[0040] The mixer 21 outputs the audio signal to the speakers 24A to 24F. The mixer 21 receives the audio signal from the playback device 22, performs signal processing such as mixing, and outputs the audio signal to a plurality of audio devices such as speakers.

[0041] The speakers 24A to 24F output the performers' voices to the second venue 20. The speakers 24A to 24F output the initial reflected sound or the rear reverberation sound for reproducing the sound field of the first venue 10. The speakers 24A to 24F output the ambient sound such as the cheers of the audience in the first venue 10 to the second venue 20.

[0042] Figure 4 2 is a block diagram showing the structure of the mixer 11. In addition, since the mixer 21 has the same structure and function as the mixer 11, Figure 4 1 shows a typical configuration of the mixer 11. The mixer 11 includes a display 101, a user I / F 102, an audio I / O (Input / Output) 103, a signal processing unit (DSP) 104, a network I / F 105, a CPU 106, a flash memory 107, and a RAM 108.

[0043] The CPU 106 is a control unit that controls the operation of the mixer 11. The CPU 106 reads a predetermined program stored in the flash memory 107 as a storage medium into the RAM 108 and executes the program, thereby performing various operations.

[0044] In addition, the program read by CPU 106 does not need to be stored in the flash memory 107 in this device. For example, the program can also be stored in a storage medium of an external device such as a server. In this case, CPU 106 can read the program from the server to RAM 108 each time and execute it.

[0045] The signal processing unit 104 is composed of a DSP for performing various signal processing. The signal processing unit 104 performs signal processing such as mixing processing and filtering processing on the sound signal input from the audio equipment such as the microphone via the audio I / O 103 or the network I / F 105. The signal processing unit 104 outputs the audio signal after the signal processing to the audio equipment such as the speaker via the audio I / O 103 or the network I / F 105.

[0046] In addition, the signal processing unit 104 can also perform sound image panning processing, initial reflected sound generation processing, and rear reverberation sound generation processing. Sound image panning processing is a process of controlling the volume of the sound signal distributed to multiple speakers 14A~14G in a manner that the sound image is localized at the position of the performer. In order to perform sound image panning processing, the CPU 106 obtains the position information of the performer via the trackers 15A~15C. The position information is information indicating a 2D or 3D coordinate based on a certain position of the first venue 10. The trackers 15A~15C are, for example, tags that transmit and receive radio waves such as Bluetooth (registered trademark). The trackers 15A~15C are installed on the performers or musical instruments. At least 3 beacons are pre-set in the first venue 10. Each beacon measures the distance between the trackers 15A~15C based on the time difference from the transmission of the radio wave to the reception of the radio wave. The CPU 106 obtains the position information of the beacons in advance and measures the distances from at least three beacons to the tags, thereby being able to uniquely determine the positions of the trackers 15A to 15C.

[0047] In this way, the CPU 106 obtains the position information of each performer, that is, the position information of the sound generated in the first venue 10, via the trackers 15A to 15C. Based on the obtained position information and the positions of the speakers 14A to 14G, the CPU 106 determines the volume of each sound signal output to the speakers 14A to 14G so that the sound image is localized at the position of the performer. The signal processing unit 104 controls the volume of each sound signal output to the speakers 14A to 14G according to the control of the CPU 106. For example, the signal processing unit 104 increases the volume of the sound signal output to the speaker at the position close to the performer, and decreases the volume of the sound signal output to the speaker at the position far from the performer. In this way, the signal processing unit 104 can localize the sound image of the performer's performance sound and singing sound at a specified position.

[0048] The initial reflected sound generation process and the rear reverberant sound generation process are processes for convolving the impulse response with the performer's voice through an FIR filter. The signal processing unit 104, for example, convolves the impulse response obtained at a predetermined venue (a venue other than the first venue 10) with the performer's voice. Thus, the signal processing unit 104 controls the sound field of the first venue 10. Alternatively, the signal processing unit 104 may further feed back the sound obtained by the microphone installed near the ceiling or wall of the first venue 10 to the speakers 14A to 14G, thereby controlling the sound field of the first venue 10.

[0049] The signal processing unit 104 outputs the performer's voice and the performer's position information to the transmission device 12. The transmission device 12 obtains the performer's voice and the performer's position information from the mixer 11.

[0050] In addition, the transmission device 12 acquires a video signal from the camera 16. The camera 16 captures each performer or the entire first venue 10, and outputs a video signal related to a live video to the transmission device 12.

[0051] Furthermore, the transmission device 12 obtains the spatial echo information of the first venue 10. The spatial echo information is information for generating indirect sound. Indirect sound is sound that reaches the audience after the sound of the sound source is reflected in the venue, and includes at least initial reflected sound and rear reverberant sound. The spatial echo information includes, for example, information indicating the size, shape, and wall material of the space of the first venue 10, and an impulse response related to the rear reverberant sound. The information indicating the size, shape, and wall material of the space is information for generating the initial reflected sound. The information for generating the initial reflected sound may also be an impulse response. The impulse response is, for example, pre-determined in the first venue 10. In addition, the spatial echo information may also be information that changes in accordance with the position of the performer. The information that changes in accordance with the position of the performer is, for example, an impulse response pre-determined for the position of each performer in the first venue 10. The transmission device 12 obtains, for example, a first impulse response when a performer's voice is generated in front of the stage, a second impulse response when a performer's voice is generated on the left side of the stage, and a third impulse response when a performer's voice is generated on the right side of the stage in the first venue 10. However, the impulse responses are not limited to three. In addition, the impulse responses do not need to be actually measured in the first venue 10, and can be obtained by simulation based on, for example, the size and shape of the space of the first venue 10, the material of the wall surface, etc.

[0052] In addition, the initial reflected sound is a reflected sound whose arrival direction is fixed, and the rear reverberant sound is a reflected sound whose arrival direction is not fixed. Compared with the initial reflected sound, the rear reverberant sound changes less due to the change in the position of the performer's voice. Therefore, the spatial echo information can be in a form composed of the impulse response of the initial reflected sound that changes according to the position of the performer and the impulse response of the rear reverberant sound that is constant regardless of the position of the performer.

[0053] In addition, the signal processing unit 104 may also obtain environmental information related to the ambient sound and output it to the transmission device 12. The ambient sound is the sound obtained by the microphones 13D to 13F as described above, including background noise, the support of the audience, applause, shouting, cheering, chorus, or noise. However, the ambient sound may also be obtained by the microphones 13A to 13C of the stage. The signal processing unit 104 outputs the sound signal related to the ambient sound to the transmission device 12 as environmental information. In addition, the environmental information may also include the position information of the ambient sound. The support such as "Come on" of the audience, the shouting of the performer's personal name, or the exclamation such as "Great" in the ambient sound are sounds that are not drowned out by the audience and can be recognized as the sound of a separate audience. The signal processing unit 104 may also obtain the position information of these separate sounds. The position information of the ambient sound can be obtained, for example, based on the sound obtained by the microphones 13D to 13F. When the above-mentioned individual sound is recognized by processing such as voice recognition, the signal processing unit 104 obtains the correlation of the sound signals of the microphones 13D to 13F, and obtains the difference in timing when the individual sounds are picked up by the microphones 13D to 13F. The signal processing unit 104 can uniquely obtain the position in the first venue 10 where the sound is generated based on the difference in timing when the sounds are picked up by the microphones 13D to 13F. In addition, the position information of the ambient sound can also be regarded as the position information of each microphone 13D to 13F.

[0054] The transmission device 12 encodes and transmits the sound source information related to the sound generated in the first venue 10 and the echo information of the space as transmission data. The sound source information includes at least the voice of the performer and may also include the position information of the performer's voice. In addition, the transmission device 12 may include the environmental information related to the environmental sound in the transmission data and transmit it. The transmission device 12 may also include the image signal related to the image of the performer in the transmission data and transmit it.

[0055] Alternatively, the transmission device 12 may transmit at least the sound source information related to the performer's voice and the performer's position information, and the environmental information related to the environmental sound as the transmission data.

[0056] Figure 5 It is a block diagram showing the structure of the transmission device 12. Figure 6It is a flowchart showing the operation of the conveying device 12.

[0057] The transmission device 12 is constituted by an information processing device such as a general personal computer, etc. The transmission device 12 includes a display 201 , a user I / F 202 , a CPU 203 , a RAM 204 , a network I / F 205 , a flash memory 206 , and a general-purpose communication I / F 207 .

[0058] The CPU 203 reads the program stored in the flash memory 206 as a storage medium into the RAM 204, thereby realizing the prescribed function. In addition, the program read by the CPU 203 does not need to be stored in the flash memory 206 in the present device. For example, the program may also be stored in the storage medium of an external device such as a server. In this case, the CPU 203 reads the program from the server into the RAM 204 each time and executes it.

[0059] The CPU 203 obtains the performer's voice and the performer's position information (sound source information) from the mixer 11 via the network I / F 205 (S11). In addition, the CPU 203 obtains the echo information of the space of the first venue 10 (S12). Furthermore, the CPU 203 obtains the environment information related to the ambient sound (S13). In addition, the CPU 203 may also obtain the image signal from the camera 16 via the general communication I / F 207.

[0060] The CPU 203 encodes and transmits data on the performer's voice and position information (sound source information) of the voice, data on spatial echo information, data on environmental information, and data on the image signal as transmission data (S14).

[0061] The playback device 22 receives the transmission data from the transmission device 12 via the Internet 5. The playback device 22 reproduces (rendering) the transmission data and provides the sound related to the performer's voice and the echo of the space to the second venue 20. Alternatively, the playback device 22 provides the performer's voice and the environmental sound included in the environmental information to the second venue 20. The playback device 22 may also provide the sound related to the echo of the space corresponding to the environmental information to the second venue 20.

[0062] Figure 7 It is a block diagram showing the structure of the playback device 22. Figure 8 It is a flowchart showing the operation of the playback device 22.

[0063] The playback device 22 is constituted by an information processing device such as a general personal computer, etc. The playback device 22 includes a display 301 , a user I / F 302 , a CPU 303 , a RAM 304 , a network I / F 305 , a flash memory 306 , and a video I / F 307 .

[0064] The CPU 303 reads the program stored in the flash memory 306 as a storage medium into the RAM 304, thereby realizing the prescribed function. In addition, the program read by the CPU 303 does not need to be stored in the flash memory 306 in the device. For example, the program may also be stored in the storage medium of an external device such as a server. In this case, the CPU 303 reads the program from the server into the RAM 304 each time and executes it.

[0065] The CPU 303 receives transmission data from the transmission device 12 via the network I / F 305 (S21). The CPU 303 decodes the transmission data into sound source information, spatial echo information, environment information, and video signals (S22), and reproduces the sound source information, spatial echo information, environment information, and video signals.

[0066] The CPU 303 causes the mixer 21 to perform a sound image shift process of the performer's voice as an example of reproduction of the sound source information (S23). The sound image shift process is a process for localizing the performer's voice at the performer's position as described above. The CPU 303 determines the volume of the sound signal distributed to the speakers 24A to 24F in such a manner that the performer's voice is localized at the position indicated by the position information included in the sound source information. The CPU 303 outputs the sound signal related to the performer's voice and the information indicating the output amount of the sound signal related to the performer's voice to the speakers 24A to 24F to the mixer 21, thereby causing the mixer 21 to perform the sound image shift process.

[0067] Thus, the audience of the second venue 20 can perceive that the sound is coming from the position of the performer. For example, the audience of the second venue 20 can also hear the sound of the performer located on the right side of the stage of the first venue 10 from the front right side in the second venue 20. In addition, the CPU 303 can also reproduce the image signal and display the live image on the display 23 via the image I / F 307. Thus, the audience of the second venue 20 can listen to the sound of the performer after the sound and image movement processing while watching the image of the performer displayed on the display 23. Thus, the audience of the second venue 20 can obtain an immersive feeling for the live performance because the visual information and the auditory information are consistent.

[0068] Furthermore, the CPU 303 causes the mixer 21 to perform indirect sound generation processing (S24) as an example of reproduction of spatial echo information. The indirect sound generation processing includes initial reflected sound generation processing and rear reverberation sound generation processing. The initial reflected sound is generated based on the voice of the performer included in the sound source information and the information indicating the size, shape, and wall material of the space of the first venue 10 included in the spatial echo information. The CPU 303 determines the arrival timing of the initial reflected sound based on the size and shape of the space, and determines the level of the initial reflected sound based on the material of the wall. More specifically, the CPU 303 obtains the coordinates of the wall surface on which the sound of the sound source is reflected based on the information on the size and shape of the space. Furthermore, the CPU 303 obtains the position of a virtual sound source (virtual sound source) that exists with the wall surface as a mirror surface relative to the position of the sound source based on the position of the sound source, the position of the wall surface, and the position of the sound receiving point. The CPU 303 obtains the delay amount of the virtual sound source based on the distance from the position of the virtual sound source to the sound receiving point. In addition, the CPU 303 obtains the level of the virtual sound source based on the information of the material of the wall. The information of the material corresponds to the energy loss when the wall is reflected. Therefore, the CPU 303 considers the energy loss for the sound signal of the sound source and obtains the level of the virtual sound source. By repeatedly performing such processing, the CPU 303 can calculate and obtain the delay amount and level of the sound related to the spatial echo. The CPU 303 outputs the calculated delay amount and level to the mixer 21. The mixer 21 convolves these delay amounts and the level tap coefficients corresponding to the level with the voice of the performer. Thus, the mixer 21 reproduces the spatial echo of the first venue 10 in the second venue 20. In addition, when the spatial echo information includes the impulse response of the initial reflected sound, the CPU 303 causes the mixer 11 to perform a process of convolving the impulse response with the voice of the performer through the FIR filter. The CPU 303 outputs the spatial echo information (impulse response) contained in the transmission data to the mixer 21. The mixer 21 convolves the spatial echo information (impulse response) received from the playback device 22 with the performer's voice. Thus, the mixer 21 reproduces the spatial echo of the first venue 10 in the second venue 20 .

[0069] Furthermore, when the spatial echo information changes in accordance with the position of the performer, the playback device 22 outputs the spatial echo information corresponding to the position of the performer to the mixer 21 based on the position information included in the sound source information. For example, when a performer at the front of the stage of the first venue 10 moves to the left side of the stage, the impulse response convolved with the performer's voice changes from the first impulse response to the second impulse response. Alternatively, when the virtual sound source is reproduced based on the information of the size and shape of the space, the delay amount and level are recalculated in accordance with the position of the performer after the move. As a result, the appropriate spatial echo corresponding to the position of the performer is also reproduced in the second venue 20.

[0070] In addition, the playback device 22 may also cause the mixer 21 to generate a spatial echo sound corresponding to the ambient sound based on the environmental information and the spatial echo information. That is, the sound involved in the spatial echo may also include a first echo sound corresponding to the performer's voice (the sound of the first sound source) and a second echo sound corresponding to the ambient sound (the sound of the second sound source). Thus, the mixer 21 reproduces the echo of the ambient sound of the first venue 10 in the second venue 20. In addition, when the environmental information includes position information, the playback device 22 may also output the spatial echo information corresponding to the position of the ambient sound to the mixer 11 based on the position information included in the environmental information. The mixer 21 reproduces the echo sound of the ambient sound based on the position of the ambient sound. For example, when the audience at the left rear of the first venue 10 moves to the right rear, the impulse response convolved with the cheering sound of the audience is changed. Alternatively, when the virtual sound source is reproduced based on the information of the size and shape of the space, the delay amount and level are recalculated according to the position of the moved audience. As described above, the spatial echo information includes the first echo information that changes according to the position of the performer's voice (the first sound source) and the second echo information that changes according to the position of the ambient sound (the second sound source), and the reproduction may include a process of generating a first echo sound based on the first echo information and a process of generating a second echo sound based on the second echo information.

[0071] In addition, the rear reverberation sound is a reflected sound whose arrival direction is not fixed. Compared with the initial reflected sound, the rear reverberation sound changes less due to the change of the sound position. Therefore, the playback device 22 can only change the impulse response of the initial reflected sound that changes according to the position of the performer, and fix the impulse response of the rear reverberation sound.

[0072] In addition, the playback device 22 can omit the indirect sound generation process and directly use the echo of the second venue 20. In addition, the indirect sound generation process can be only the initial reflected sound generation process. The rear reverberation sound can also directly use the echo of the second venue 20. Alternatively, the mixer 21 can also strengthen the control of the second venue 20 by further feeding back the sound obtained by the unillustrated microphone set near the ceiling and wall of the second venue 20 to the speakers 24A to 24F.

[0073] Furthermore, the CPU 303 of the playback device 22 performs playback processing of the ambient sound based on the environmental information (S25). The environmental information includes sound signals of background noise, audience support, applause, shouting, cheering, chorus, or noise. The CPU 303 outputs these sound signals to the mixer 21. The mixer 21 outputs the sound signals received from the playback device 22 to the speakers 24A to 24F.

[0074] When the environmental information includes the position information of the environmental sound, the CPU 303 causes the mixer 21 to perform the localization processing of the environmental sound by the sound image shift processing. In this case, the CPU 303 determines the volume of the sound signal allocated to the speakers 24A to 24F in such a way that the environmental sound is localized at the position information included in the environmental information. The CPU 303 causes the mixer 21 to perform the sound image shift processing by outputting the sound signal of the environmental sound and the information indicating the output amount of the sound signal related to the environmental sound to the speakers 24A to 24F to the mixer 21. In addition, the same is true when the position information of the environmental sound is the position information of each microphone 13D to 13F. The CPU 303 determines the volume of the sound signal allocated to the speakers 24A to 24F in such a way that the environmental sound is localized at the position of the microphone. Each microphone 13D to 13F picks up multiple environmental sounds (second sound sources) such as background noise, applause, chorus, or cheers such as "wow", and noise. The sound of each sound source reaches the microphone with a predetermined delay and level. That is, background noise, applause, chorus, or cheers such as "wow", noise, etc. also reach the microphone as independent sound sources with a predetermined delay and level (information for localizing the sound source). CPU 303 performs sound image movement processing in a manner that localizes the sound picked up by the microphone at the position of the microphone, thereby being able to easily reproduce the localization of the independent sound source.

[0075] In addition, the CPU 303 may also process the sounds that cannot be identified as the sounds of a single listener and that are emitted by multiple listeners simultaneously, that is, by causing the mixer 21 to perform effect processing such as reverberation, so as to perceive the expansion of space. For example, background noise, applause, chorus, or cheers such as "wow", noise, etc. are sounds that resound throughout the entire live venue. The CPU 303 causes the mixer 21 to perform effect processing to perceive the expansion of space on these sounds.

[0076] The playback device 22 may provide the ambient sound based on the above-described ambient information to the second venue 20. Thus, the audience of the second venue 20 can watch and hear the live performance with a higher sense of presence, just as they watch and hear the live performance at the first venue 10.

[0077] As described above, the on-site data transmission system 1 of the present embodiment transmits the sound source information related to the sound generated in the first venue 10 and the spatial echo information as transmission data, reproduces the transmission data, and provides the sound related to the sound source information and the sound related to the spatial echo to the second venue 20. Thus, the sense of presence of the on-site venue can be provided to the transmission destination venue as well.

[0078] In addition, the live data transmission system 1 transmits the first sound source information related to the sound (e.g., the voice of the performer) of the first sound source occurring at a first place (e.g., the stage) of the first venue 10 and the position information of the first sound source, and the second sound source information related to the second sound source (e.g., the ambient sound) occurring at a second place (e.g., the place where the audience is located) of the first venue 10 as transmission data, reproduces the transmission data, and provides the sound of the first sound source and the sound of the second sound source to which the positioning processing based on the position information of the first sound source has been performed to the second venue. Thus, the sense of presence of the live venue can be provided to the transmission target venue as well.

[0079] then, Fig. 9 This is a block diagram showing the configuration of a field data transmission system 1A according to Modification 1. Fig.10 FIG. 2 is a schematic plan view of a second venue 20 of an on-site data transmission system 1A according to Modification 1. Figure 1 and Figure 3 The same reference numerals are used to denote common structures, and description thereof will be omitted.

[0080] A plurality of microphones 25A to 25C are installed in the second venue 20 of the live data transmission system 1A. Microphone 25A is installed on the left side of the front and back center facing the stage 80 of the second venue 20, and microphone 25B is installed in the rear center of the second venue 20. Microphone 25C is installed on the right side of the front and back center of the second venue 20.

[0081] Microphones 25A to 25C acquire the ambient sound of second venue 20. Mixer 21 outputs the sound signal of the ambient sound as environmental information to playback device 22. Environmental information may also include position information of the ambient sound. The position information of the ambient sound can be obtained, for example, from the sound acquired by microphones 25A to 25C as described above.

[0082] The playback device 22 transmits the environmental information related to the environmental sound generated in the second venue 20 to other venues as the third sound source. For example, the playback device 22 feeds back the environmental sound generated in the second venue 20 to the first venue 10. As a result, the performers on the stage of the first venue 10 can hear sounds, applause, cheers, etc. other than those of the audience of the first venue 10, and can perform live performances in an environment full of a sense of presence. In addition, the audience in the first venue 10 can also hear sounds, applause, cheers, etc. of the audience in other venues, and can watch and listen to the live performance in an environment full of a sense of presence.

[0083] Furthermore, if the playback device at the other venue reproduces the transmitted data and provides the sound of the first venue to the other venue, and provides the ambient sound occurring at the second venue 20 to the other venue, the audience at the other venue will also be able to hear the voices, applause, cheers, etc. of multiple audiences, and will be able to view and listen to the live performance in an immersive environment.

[0084] then, Fig.11 2 is a block diagram showing the structure of a field data transmission system 1B according to a second modification. Figure 1 The same reference numerals are used to denote common structures, and description thereof will be omitted.

[0085] In the live data transmission system 1B, the transmission device 12 is connected to the AV receiver 32 of the third venue 20A via the Internet 5. The AV receiver 32 is connected to a display 33, a plurality of speakers 34A to 34F, and a microphone 35. The third venue 20A is, for example, a home of a certain listener. The AV receiver 32 is an example of a playback device. The user of the AV receiver 32 is an audience who remotely watches and listens to the live performance of the first venue 10.

[0086] Fig.12 4 is a block diagram showing the structure of the AV receiver 32. The AV receiver 32 includes a display 401, a user I / F 402, an audio I / O (Input / Output) 403, a signal processing unit (DSP) 404, a network I / F 405, a CPU 406, a flash memory 407, a RAM 408, and a video I / F 409.

[0087] The CPU 406 is a control unit that controls the operation of the AV receiver 32. The CPU 406 reads a predetermined program stored in the flash memory 407 as a storage medium into the RAM 408 and executes the program, thereby performing various operations.

[0088] In addition, the program read by the CPU 406 does not need to be stored in the flash memory 407 in the device. For example, the program can also be stored in a storage medium of an external device such as a server. In this case, the CPU 406 reads the program from the server to the RAM 408 each time and executes it.

[0089] The signal processing unit 404 is composed of a DSP for performing various signal processing. The signal processing unit 404 performs signal processing on the audio signal input via the audio I / O 403 or the network I / F 405. The signal processing unit 404 outputs the processed audio signal to an audio device such as a speaker via the audio I / O 403 or the network I / F 405.

[0090] The AV receiver 32 performs the same processing as that performed by the mixer 21 and the playback device 22. The CPU 406 receives the transmission data from the transmission device 12 via the network I / F 405. The CPU 406 reproduces the transmission data and provides the sound related to the performer's voice and the echo of the space to the third venue 20A. Alternatively, the CPU 406 reproduces the transmission data and provides the ambient sound generated in the first venue 10 to the third venue 20A. Alternatively, the CPU 406 may also reproduce the transmission data and display the live image on the display 33 via the image I / F 307.

[0091] The signal processing unit 404 performs a sound image shift process of the performer's voice. In addition, the signal processing unit 404 performs a process of generating indirect sound. Alternatively, the signal processing unit 404 may perform a sound image shift process of the ambient sound.

[0092] Thus, the AV receiver 32 can provide the sense of presence of the first site 10 also to the third site 20A.

[0093] In addition, the AV receiver 32 acquires the ambient sound of the third venue 20A (sounds such as cheers, applause, or shouts of the audience) via the microphone 35. The AV receiver 32 transmits the ambient sound of the third venue 20A to other devices. For example, the AV receiver 32 feeds back the ambient sound of the third venue 20A to the first venue 10.

[0094] In this way, if the sounds from multiple audiences are fed back to the first venue 10, the performers on the stage of the first venue 10 can hear the support, applause, cheers, etc. of multiple audiences other than the audience of the first venue 10, and can perform live performances in an environment full of a sense of presence. In addition, the audience in the first venue 10 can also hear the support, applause, cheers, etc. of multiple remote audiences, and can watch and listen to the live performance in an environment full of a sense of presence.

[0095] Alternatively, the AV receiver 32 may display icon images such as "support", "applause", "shout" and "noise" on the display 401, and receive selection operations for these icon images from the audience via the user I / F 402, thereby receiving the audience's response. If the AV receiver 32 receives the selection operations for these responses, it may generate a sound signal corresponding to each response and send it to other devices as environmental information.

[0096] Alternatively, the AV receiver 32 may also transmit information indicating the type of ambient sound such as the support, applause, or shouting of the audience as the ambient information. In this case, the device on the receiving side (e.g., the transmission device 12 and the mixer 11) generates a corresponding sound signal based on the ambient information, and provides the sound such as the support, applause, or shouting of the audience to the venue. In this way, the ambient information is not a sound signal of the ambient sound, but information indicating the sound to be generated, and the pre-recorded ambient sound can be played by the transmission device 12 and the mixer 11.

[0097] In addition, the environmental information of the first venue 10 may not be the environmental sound occurring in the first venue 10, but the pre-recorded environmental sound. In this case, the transmission device 12 transmits the information indicating the sound to be generated as the environmental information. The playback device 22 or the AV receiver 32 plays the corresponding environmental sound based on the environmental information. In addition, the background noise and noise in the environmental information may be the recorded sound, and the other environmental sounds (such as the audience's support, applause, shouting, etc.) may be the sound occurring in the first venue 10.

[0098] The AV receiver 32 may also receive the position information of the audience via the user I / F 402. The AV receiver 32 displays an image imitating a plan view or a perspective view of the first venue 10 on the display 401 or the display 33, and receives the position information (for example, referring to the position information of the first venue 10) from the audience via the user I / F 402. Fig.16). The position information is information for specifying an arbitrary position in the first venue 10. The AV receiver 32 transmits the received position information of the listener to the first venue 10. The transmission device 12 and the mixer 11 of the first venue perform processing for localizing the ambient sound of the third venue 20A at the specified position based on the ambient sound of the third venue 20A and the position information of the listener received from the AV receiver 32.

[0099] In addition, the AV receiver 32 may also change the content of the sound image movement processing based on the position information received from the user. For example, if the audience specifies the position immediately before the stage of the first venue 10, the AV receiver 32 sets the localization position of the performer's voice to the position immediately before the audience and performs the sound image movement processing. As a result, the audience in the third venue 20A can obtain the same sense of presence as if they were immediately before the stage of the first venue 10.

[0100] The voices of the audience at the third venue 20A may be sent to the second venue 20 instead of the first venue 10, or may be sent to other venues. For example, the voices of the audience at the third venue 20A may be sent only to a friend's house (the fourth venue). The audience at the fourth venue can listen to the voices of the audience at the third venue 20A while watching and listening to the live performance at the first venue 10. In addition, the unillustrated playback device at the fourth venue may also send the voices of the audience at the fourth venue to the third venue 20A. In this case, the audience at the third venue 20A can listen to and listen to the live performance at the first venue 10 while listening to the voices of the audience at the fourth venue. Thus, the audience at the third venue 20A and the audience at the fourth venue can listen to and listen to the live performance at the first venue 10 while talking to each other.

[0101] Fig.13 FIG. 1 is a block diagram showing the structure of a field data transmission system 1C according to a third modification. Figure 1 The same reference numerals are used to denote common structures, and description thereof will be omitted.

[0102] In the live data transmission system 1C, the transmission device 12 is connected to the terminal 42 of the 5th venue 20B via the Internet 5. The terminal 42 is connected to the earphone 43. The 5th venue 20B is, for example, the home of a certain listener. However, in the case where the terminal 42 is mobile, the 5th venue 20B can also be various places such as a coffee shop, a car, or a public transportation facility. In this case, all places can become the 5th venue 20B. The terminal 42 is an example of a playback device. The user of the terminal 42 becomes an audience who remotely watches and listens to the live performance of the first venue 10. In this case, the terminal 42 also reproduces the transmission data, and provides the sound related to the sound source information and the sound related to the echo of the space to the second venue (in this example, the 5th venue 20B) via the earphone 43.

[0103] Fig.14 5 is a block diagram showing the structure of the terminal 42. The terminal 42 is an information processing device such as a personal computer, a smartphone or a tablet computer. The terminal 42 has a display 501, a user I / F 502, a CPU 503, a RAM 504, a network I / F 505, a flash memory 506, an audio I / O (Input / Output) 507 and a microphone 508.

[0104] The CPU 503 is a control unit that controls the operation of the terminal 42. The CPU 503 reads a predetermined program stored in the flash memory 506 as a storage medium into the RAM 504 and executes the program, thereby performing various operations.

[0105] In addition, the program read by the CPU 503 does not need to be stored in the flash memory 506 in the device. For example, the program can also be stored in a storage medium of an external device such as a server. In this case, the CPU 503 reads the program from the server to the RAM 504 each time and executes it.

[0106] The CPU 503 performs signal processing on the audio signal input via the network I / F 505. The CPU 503 outputs the audio signal after the signal processing to the headphone 43 via the audio I / O 507.

[0107] The CPU 503 receives the transmission data from the transmission device 12 via the network I / F 505. The CPU 503 reproduces the transmission data, and provides the audience in the fifth venue 20B with the sound related to the performer's voice and the echo of the space.

[0108] Specifically, the CPU 503 convolves the sound signal related to the performer's voice with the head transfer function (hereinafter referred to as HRTF) to perform sound image localization processing (binaural processing) in a manner that the performer's voice is localized at the performer's position. HRTF corresponds to the transfer function between a specified position and the listener's ears. HRTF is a transfer function that expresses the size, arrival time, and frequency characteristics of the sound that reaches the left and right ears respectively from a sound source at a certain position. The CPU 503 convolves the HRTF with the sound signal of the performer's voice based on the performer's position. As a result, the performer's voice is localized at a position corresponding to the position information.

[0109] In addition, CPU 503 performs binaural processing of indirect sound generation by convolving the sound signal of the performer's voice with the HRTF corresponding to the spatial echo information. CPU 503 localizes the initial reflected sound and the rear echo sound by convolving the HRTF that reaches the left and right ears respectively from the positions of the virtual sound sources corresponding to the initial reflected sounds contained in the spatial echo information. However, the rear echo sound is a reflected sound whose arrival direction is not fixed. Therefore, CPU 503 may not perform localization processing on the rear echo sound, but perform effect processing such as reverberation. In addition, CPU 503 may also perform digital filtering processing (headphone inverse characteristic processing) to reproduce the inverse characteristics of the acoustic characteristics of the headphones 43 used by the audience.

[0110] In addition, the CPU 503 reproduces the environmental information in the transmission data and provides the environmental sound generated in the first venue 10 to the audience in the fifth venue 20B. When the environmental information includes the position information of the environmental sound, the CPU 503 performs localization processing based on HRTF and performs effect processing on the sound whose arrival direction is not fixed.

[0111] In addition, the CPU 503 may also reproduce the image signal in the transmission data and display the live image on the display 501 .

[0112] Thus, the terminal 42 can also provide the sense of presence of the first site 10 to the audience at the fifth site 20B.

[0113] In addition, the terminal 42 obtains the voice of the audience in the fifth venue 20B via the microphone 508. The terminal 42 sends the voice of the audience to other devices. For example, the terminal 42 feeds back the voice of the audience to the first venue 10. Alternatively, the terminal 42 may also display icon images such as "support", "applause", "shout" and "noise" on the display 501, receive selection operations for these icon images from the audience via the user I / F 502, and receive responses. The terminal 42 generates a sound corresponding to the received response, and sends the generated sound to other devices as environmental information. Alternatively, the terminal 42 may also send information indicating the type of environmental sound such as support, applause, or shouting from the audience as environmental information. In this case, the device on the receiving side (for example, the transmission device 12 and the mixer 11) generates a corresponding sound signal based on the environmental information, and provides the sound such as support, applause, or shouting from the audience to the venue.

[0114] In addition, the terminal 42 may receive the position information of the audience via the user I / F 502. The terminal 42 transmits the received position information of the audience to the first site 10. The transmission device 12 and the mixer 11 of the first site perform processing to localize the voice of the audience at the designated position based on the voice and position information of the audience of the third site 20A received from the AV receiver 32.

[0115] In addition, the terminal 42 may also change the HRTF based on the position information received from the user. For example, if the audience specifies the position just before the stage of the first venue 10, the terminal 42 sets the position of the performer's voice to the position just before the audience, and convolves the HRTF that localizes the performer's voice to the position. As a result, the audience in the fifth venue 20B can obtain the same sense of presence as if they were just before the stage of the first venue 10.

[0116] The voices of the audience at the fifth venue 20B may be sent to the second venue 20 instead of the first venue 10, or may be sent to other venues. Similarly to the above, the voices of the audience at the fifth venue 20B may be sent only to the friend's house (the fourth venue). Thus, the audience at the fifth venue 20B and the audience at the fourth venue can watch and listen to the live performance at the first venue 10 while talking to each other.

[0117] In addition, in the on-site data transmission system of this embodiment, multiple users can also specify the same position. For example, multiple users can respectively specify the position just before the stage of the first venue 10. In this case, each audience can obtain a sense of presence as if they were just before the stage. Thus, for one position (the seat of the venue), multiple audiences can watch and listen to the performance of the performer with the same sense of presence. In this case, the on-site operator can provide services that exceed the number of spectators that can be accommodated in the actual space.

[0118] Fig.15 1 is a block diagram showing the structure of a field data transmission system 1D according to Modification 4. Figure 1 The same reference numerals are used to denote common structures, and description thereof will be omitted.

[0119] The on-site data transmission system 1D further includes a server 50 and a terminal 55. The terminal 55 is installed in the sixth venue 10A. The server 50 is an example of a transmission device, and the hardware structure of the server 50 is the same as that of the transmission device 12. The hardware structure of the terminal 55 is the same as that of the transmission device 12. Fig.14 The structures of the terminals 42 shown are identical.

[0120] The sixth venue 10A is a residence or the like of a performer who performs a performance such as a musical instrument remotely. The performer in the sixth venue 10A performs a performance such as a musical instrument or a singing instrument in accordance with the performance or singing instrument in the first venue. The terminal 55 transmits the sound of the performer in the sixth venue 10A to the server 50. In addition, the terminal 55 may also shoot the performer in the sixth venue 10A with a camera (not shown) and transmit the image signal to the server 50.

[0121] The server 50 transmits transmission data including the voices of performers in the first venue 10, the voices of performers in the sixth venue 10A, echo information of the space of the first venue 10, environmental information of the first venue 10, live images of the first venue 10, and images of performers in the sixth venue 10A.

[0122] In this case, the playback device 22 reproduces the transmitted data and provides the voices of the performers in the first venue 10, the voices of the performers in the sixth venue 10A, the echo of the space of the first venue 10, the ambient sound of the first venue 10, the live video of the first venue 10, and the video of the performers in the sixth venue 10A to the second venue 20. For example, the playback device 22 displays the video of the performers in the sixth venue 10A superimposed on the live video of the first venue 10.

[0123] The sound of the performer in the sixth venue 10A may not be localized, or may be localized to a position that matches the image displayed on the display. For example, if the performer in the sixth venue 10A is displayed on the right side of the live image, the sound of the performer in the sixth venue 10A is localized to the right side.

[0124] In addition, the performer of the sixth venue 10A or the transmitter of the transmission data may specify the position of the performer. In this case, the transmission data includes the position information of the performer of the sixth venue 10A. The playback device 22 locates the sound of the performer of the sixth venue 10A based on the position information of the performer of the sixth venue 10A.

[0125] The image of the performer in the sixth venue 10A is not limited to the image captured by the camera, and for example, a two-dimensional image or an image of a virtual character (virtual image) composed of 3D modeling may be transmitted as the image of the performer in the sixth venue 10A.

[0126] In addition, the transmission data may also include recording data. In addition, the transmission data may also include video data. For example, the transmission device may transmit the transmission data including the voice of the performer of the first venue 10, the recording data, the echo information of the space of the first venue 10, the environmental information of the first venue 10, the live image of the first venue 10, and the video data. In this case, the playback device reproduces the transmission data and provides the voice of the performer of the first venue 10, the sound related to the recording data, the echo of the space of the first venue 10, the environmental sound of the first venue 10, the live image of the first venue 10, and the image related to the video data to other venues. The playback device 22 displays the image of the performer corresponding to the video data superimposed on the live image of the first venue 10.

[0127] In addition, the transmission device may also determine the type of musical instrument when recording the sound involved in the recording data. In this case, the transmission device includes information indicating the type of musical instrument determined as the recording data in the transmission data and transmits it. The playback device generates an image of the corresponding musical instrument based on the information indicating the type of the musical instrument. The playback device may also display the image of the musical instrument superimposed on the live image of the first venue 10.

[0128] In addition, the transmission data does not need to superimpose the image of the performer in the sixth venue 10A on the live image of the first venue 10. For example, the transmission data may also transmit the image of each performer and the background image in the first venue 10 and the sixth venue 10A as separate data. In this case, the transmission data includes information indicating the display position of each image. The playback device reproduces the image of each performer based on the information indicating the display position.

[0129] In addition, the background image is not limited to the image of the venue where the live performance is actually performed, such as the first venue 10. The background image may be an image of a venue different from the venue where the live performance is performed.

[0130] Furthermore, the spatial echo information included in the transmission data does not need to correspond to the spatial echo of the first venue 10. For example, the spatial echo information may be virtual spatial information (information indicating the size, shape, wall material, etc. of the space of each venue, or an impulse response indicating the transfer function of each venue) for virtually reproducing the spatial echo of the venue corresponding to the background image. The impulse response of each venue may be measured in advance, or may be obtained by simulation based on the size, shape, wall material, etc. of the space of each venue.

[0131] Furthermore, the environmental information may also be changed to content corresponding to the background image. For example, in the case of a background image of a conference hall, the environmental information includes the sounds of multiple listeners' cheers, applause, and cheers. In addition, an outdoor venue includes background noise that is different from that of an indoor venue. In addition, the echo of the environmental sound also changes in accordance with the echo information of the above-mentioned space. In addition, the environmental information may also include information indicating the number of spectators and information indicating the degree of crowding (the density of people). The playback device increases or decreases the number of sounds of the audience's cheers, applause, and cheers based on the information indicating the number of spectators. In addition, the playback device increases or decreases the volume of the audience's cheers, applause, and cheers based on the information indicating the degree of crowding.

[0132] Alternatively, the environmental information may be changed according to the performer. For example, when a performer with many female fans performs live, the sounds of the audience's support, shouting, cheering, etc. included in the environmental information are changed to female voices. The environmental information may include sound signals of the audience's voices, and may also include information indicating attributes of the audience such as the ratio of men to women or the ratio of ages. The playback device changes the sound quality of the audience's support, applause, cheering, etc. based on the information indicating the attributes.

[0133] In addition, the audience in each venue can also specify the background image and the echo information of the space. The audience in each venue specifies the background image and the echo information of the space using the user I / F of the playback device.

[0134] Fig.16 700 is a diagram showing an example of a live video 700 displayed by a playback device of each venue. The live video 700 is composed of an image obtained by shooting the first venue 10 or other venues, or a virtual image (computer graphics) corresponding to each venue. The live video 700 is displayed on the display of the playback device. The background of the venue, the stage, performers including musical instruments, and the images of the audience in the venue are displayed in the live video 700. The background of the venue, the stage, performers including musical instruments, and the images of the audience in the venue can be all actually shot images, or virtual images. In addition, only the background image can be an actually shot image, and the other images are virtual images. In addition, an icon image 751 and an icon image 752 for specifying a space are displayed in the live video 700. The icon image 751 is an image for specifying the space of a certain venue, namely, stage A (Stage A) (for example, the first venue 10), and the icon image 752 is an image for specifying the space of another venue, namely, stage B (Stage B) (for example, other concert halls, etc.). Furthermore, an audience image 753 for specifying the position of the audience is displayed on the live video 700 .

[0135] The audience using the playback device specifies a desired space by specifying either icon image 751 or icon image 752 using the user I / F of the playback device. The transmission device includes the background image corresponding to the specified space and the echo information of the space in the transmission data and transmits it. Alternatively, the transmission device may include a plurality of background images and the echo information of the space in the transmission data and transmit it. In this case, the playback device reproduces the background image and the echo information of the space corresponding to the space specified by the audience in the received transmission data.

[0136] exist Fig.16 In the example of , icon image 751 is specified. The playback device displays the background image (for example, the image of the first venue 10) corresponding to Stage A of icon image 751, and plays the sound related to the echo of the space corresponding to the specified Stage A. If the audience specifies icon image 752, the playback device switches to display the background image of another space, Stage B, corresponding to icon image 752, and plays the sound related to the echo of the other corresponding space based on the virtual space information corresponding to Stage B.

[0137] This allows listeners of each playback device to obtain a sense of presence as if they were viewing a live performance in a desired space.

[0138] In addition, the audience of each playback device can specify a desired position in the venue by moving the audience image 753 in the live video 700. The playback device performs positioning processing based on the position specified by the user. For example, if the audience moves the audience image 753 to the position just before the stage, the playback device sets the position of the performer's voice to the position just before the audience, and performs positioning processing to position the performer's voice at the position. As a result, the audience of each playback device can obtain a sense of presence as if they were just before the stage.

[0139] In addition, as described above, if the position of the sound source and the position of the listener (the position of the sound receiving point) change, the sound related to the spatial echo also changes. The playback device can also calculate the initial reflected sound when the space changes, the position of the sound source changes, or the position of the sound receiving point changes. Therefore, even if the impulse response is not measured in the actual space, the playback device can obtain the sound related to the spatial echo based on the virtual space information. Therefore, the playback device can realize the echo generated in the space including the actual space with high accuracy.

[0140] For example, mixer 11 can function as a transmission device, and mixer 21 can function as a playback device. In addition, the playback device does not need to be set up in each venue. Fig.15 The server 50 shown may also reproduce the transmission data and transmit the audio signal after signal processing to the terminals of each venue, etc. In this case, the server 50 functions as a playback device.

[0141] The sound source information may also include information indicating the performer's posture (for example, the performer's left and right orientation). The playback device may also adjust the volume or frequency characteristics based on the performer's posture information. For example, the playback device uses the performer's orientation facing directly opposite as a reference, and the greater the left and right orientation, the lower the volume. In addition, the playback device may also perform a process in which the high-frequency band is attenuated more significantly than the low-frequency band as the left and right orientation increases. As a result, the sound changes in accordance with the performer's posture, so that the audience can view and listen to a more immersive live performance.

[0142] then, Fig.17 is a block diagram showing an application example of signal processing performed by a playback device. Fig.13 The playback device (in Fig.13 In the example, the terminal 42) functionally includes a musical instrument model processing unit 551, an amplifier model processing unit 552, a speaker model processing unit 553, a space model processing unit 554, a binaural processing unit 555, and a headphone inverse characteristic processing unit 556.

[0143] The musical instrument model processing unit 551, the amplifier model processing unit 552, and the speaker model processing unit 553 perform signal processing to give the acoustic characteristics of the acoustic device to the sound signal related to the performance sound. The first digital signal processing model used for performing the signal processing is included in the sound source information transmitted by the transmission device 12, for example. The first digital signal processing model is a digital filter that simulates the acoustic characteristics of the musical instrument, the acoustic characteristics of the amplifier, and the acoustic characteristics of the speaker. The first digital signal processing model is pre-made by the manufacturer of the musical instrument, the manufacturer of the amplifier, and the manufacturer of the speaker through simulation, etc. The musical instrument model processing unit 551, the amplifier model processing unit 552, and the speaker model processing unit 553 perform digital filtering processing that simulates the acoustic characteristics of the musical instrument, the acoustic characteristics of the amplifier, and the acoustic characteristics of the speaker, respectively. In addition, when the musical instrument is an electronic musical instrument such as a synthesizer, the musical instrument model processing unit 551 inputs note event data (information indicating the timing of sound to be sounded, pitch, etc.) instead of the sound signal, and generates a sound signal having the acoustic characteristics of an electronic musical instrument such as a synthesizer.

[0144] Thus, the playback device can reproduce the acoustic characteristics of any musical instrument, etc. Fig.16 In the video, a live video 700 of a virtual image (computer graphics) is displayed. Here, the audience using the playback device can also use the user I / F of the playback device to change to another image of a virtual musical instrument. When the audience changes the musical instrument displayed on the live video 700 to the image of another musical instrument, the musical instrument model processing unit 551 of the playback device performs signal processing corresponding to the first digital signal processing model, and the first digital signal processing model corresponds to the changed musical instrument. As a result, the playback device outputs a sound that reproduces the acoustic characteristics of the musical instrument displayed on the live video 700.

[0145] Similarly, the listeners using the playback device can also use the user I / F of the playback device to change the type of amplifier and the type of speaker to different types. The amplifier model processing unit 552 and the speaker model processing unit 553 perform digital filtering processing that simulates the acoustic characteristics of the amplifier and the acoustic characteristics of the speaker of the changed type. In addition, the speaker model processing unit 553 can also simulate the acoustic characteristics of each direction of the speaker. In this case, the listeners using the playback device can also use the user I / F of the playback device to change the direction of the speaker. The speaker model processing unit 553 performs digital filtering processing according to the changed direction of the speaker.

[0146] The space model processing unit 554 is a second digital signal processing model that reproduces the acoustic characteristics of the room at the live venue (for example, the echo of the above space). The second digital signal processing model can also be obtained by using test sounds in the actual live venue, for example. Alternatively, the second digital signal processing model can also calculate the delay amount and level of the virtual sound source based on the virtual space information (information indicating the size, shape, and wall material of each venue) as described above.

[0147] If the position of the sound source and the position of the listener (the position of the sound receiving point) change, the sound related to the echo of the space also changes. The playback device can also calculate the delay amount and level of the virtual sound source when the space changes, the position of the sound source changes, and the position of the sound receiving point changes. Therefore, even if the impulse response and other measurements are not performed in the actual space, the playback device can obtain the sound related to the echo of the space based on the virtual space information. Therefore, the playback device can realize the echo generated in the space including the actual space with high accuracy.

[0148] In addition, the virtual space information may also include information on the position and material of structures such as columns (obstacles on the sound). In the process of locating the sound source and generating the indirect sound, when there are obstacles in the paths of the direct sound and the indirect sound arriving from the sound source, the playback device reproduces the reflection, occlusion, and diffraction phenomena caused by the obstacles.

[0149] Fig.18 Schematic diagram showing the path of sound that is reflected from the wall surface from the sound source 70 and reaches the sound collecting point 75 . Fig.18 The sound source 70 shown may also be any one of a performance sound (first sound source) or an ambient sound (second sound source). Based on the position of the sound source 70, the position of the wall, and the position of the sound receiving point 75, the playback device calculates the position of a virtual sound source 70A that exists with the wall as a mirror relative to the position of the sound source 70. Furthermore, the playback device calculates the delay of the virtual sound source 70A based on the distance from the virtual sound source 70A to the sound receiving point 75. In addition, the playback device calculates the level of the virtual sound source 70A based on information about the material of the wall. Furthermore, the playback device calculates the level of the virtual sound source 70A based on information about the material of the wall. Fig.18 As shown in FIG. 1 , when there is an obstacle 77 on the path from the virtual sound source 70A to the sound receiving point 75, the frequency characteristics caused by diffraction through the obstacle 77 are obtained. Diffraction, for example, attenuates the sound in the high frequency band. Fig.18 As shown, when there is an obstacle 77 on the path from the position of the virtual sound source 70A to the sound collecting point 75, equalization processing is performed to reduce the level of the high frequency band. The frequency characteristics caused by diffraction may also be included in the virtual space information.

[0150] In addition, the playback device may also set a new second virtual sound source 77A and a third virtual sound source 77B at the left and right positions of the obstacle 77. The second virtual sound source 77A and the third virtual sound source 77B correspond to the new sound source generated by diffraction. The second virtual sound source 77A and the third virtual sound source 77B are sounds that are respectively given frequency characteristics generated by diffraction to the sound of the virtual sound source 70A. The playback device recalculates the delay amount and the level based on the positions of the second virtual sound source 77A and the third virtual sound source 77B and the position of the sound receiving point 75. In this way, the diffraction phenomenon of the obstacle 77 can be reproduced.

[0151] The playback device may also calculate the delay amount and level of the sound of the virtual sound source 70A that is reflected by the obstacle 77 and further reflected by the wall surface and reaches the sound receiving point 75. In addition, the playback device may eliminate the virtual sound source 70A when it is determined that the virtual sound source 70A is blocked by the obstacle 77. Information determining whether or not the virtual sound source is blocked may also be included in the virtual space information.

[0152] By performing the above-described processing, the playback device performs first digital signal processing that expresses the acoustic characteristics of the audio equipment and second digital signal processing that expresses the acoustic characteristics of the room, thereby generating sound of the sound source and sound related to the spatial echo.

[0153] The binaural processing unit 555 convolves the sound signal with a head transfer function (hereinafter referred to as HRTF) to perform sound image localization processing of the sound source and various indirect sounds. The headphone inverse characteristic processing unit 556 performs digital filtering processing to reproduce the inverse characteristics of the acoustic characteristics of the headphone used by the listener.

[0154] Through the above processing, the user can obtain a sense of presence as if he or she is viewing a live performance in a desired space and using desired audio equipment.

[0155] In addition, the playback device does not need to have Fig.17 The musical instrument model processing unit 551, the amplifier model processing unit 552, the speaker model processing unit 553, and the space model processing unit 554 are all shown. The playback device can perform signal processing using at least one digital signal processing model. In addition, the playback device can perform signal processing on a certain sound signal (for example, the voice of a certain performer) using one digital signal processing model, and can also perform signal processing on multiple sound signals using one digital signal processing model respectively. The playback device can perform signal processing on a certain sound signal (for example, the voice of a certain performer) using multiple digital signal processing models, and can also perform signal processing on multiple sound signals using multiple digital signal processing models. The playback device can also perform signal processing on ambient sound using a digital signal processing model.

[0156] The description of the present embodiment is illustrative in all aspects and is not restrictive. The scope of the present invention is not represented by the above-mentioned embodiment but by the claims. Moreover, the scope of the present invention includes all changes within the meaning equivalent to the claims and the scope.

[0157] Description of the label

[0158] 1. 1A, 1B, 1C, 1D…Field data transmission system

[0159] 5…Internet

[0160] 10…Site 1

[0161] 10A…Site 6

[0162] 11…Mixer

[0163] 12…Transmission device

[0164] 13A~13F…Microphone

[0165] 14A~14G…Speaker

[0166] 15A~15C…Tracker

[0167] 16…Camera

[0168] 20…Site 2

[0169] 20A…Site 3

[0170] 20B…Site 5

[0171] 21…Mixer

[0172] 22…Playback device

[0173] 23…Display

[0174] 24A~24F…Speaker

[0175] 25A~25C…Microphone

[0176] 32…AV Receiver

[0177] 33…Display

[0178] 34A…Speaker

[0179] 35…Microphone

[0180] 42…Terminal

[0181] 43…Headphones

[0182] 50…Server

[0183] 55…Terminal

[0184] 101…Display

[0185] 102…User I / F

[0186] 103…Audio I / O

[0187] 104…Signal Processing Unit

[0188] 105…Network I / F

[0189] 106…CPU

[0190] 107… Flash memory

[0191] 108…RAM

[0192] 201…Display

[0193] 202…User I / F

[0194] 203…CPU

[0195] 204…RAM

[0196] 205…Network I / F

[0197] 206… Flash memory

[0198] 207...General communication I / F

[0199] 301…Display

[0200] 302…User I / F

[0201] 303…CPU

[0202] 304…RAM

[0203] 305…Network I / F

[0204] 306… Flash memory

[0205] 307…Image I / F

[0206] 401... Display

[0207] 402…User I / F

[0208] 403…Audio I / O

[0209] 404…Signal Processing Department

[0210] 405…Network I / F

[0211] 406…CPU

[0212] 407…Flash

[0213] 408…RAM

[0214] 409…Image I / F

[0215] 501…Display

[0216] 503…CPU

[0217] 504…RAM

[0218] 505…Network I / F

[0219] 506…Flash

[0220] 507…Audio I / O

[0221] 508…Microphone

[0222] 700…Live Video

Claims

1. A method for transmitting data on site, wherein: transmitting, as transmission data, first sound source information related to a sound of a first sound source occurring at a first location of a first venue and position information of the first sound source, and second sound source information related to a second sound source including ambient sound occurring at a second location of the first venue, The transmission data is reproduced, and the sound of the first sound source and the sound of the second sound source that have been subjected to positioning processing based on the position information of the first sound source are provided to the second venue, wherein the transmission data includes echo information of the space of the first venue, and the reproduction includes processing of providing sound related to the echo of the space to the second venue, The sound related to the echo in the space includes a first echo sound corresponding to the sound of the first sound source and a second echo sound corresponding to the sound of the second sound source. The spatial echo information includes first echo information that changes according to a position of the first sound source and second echo information that changes according to a position of the second sound source, and the reproduction includes a process of generating the first echo sound based on the first echo information and a process of generating the second echo sound based on the second echo information.

2. The on-site data transmission method according to claim 1, wherein: Environmental information related to the environmental sound of the second venue is transmitted to outside the second venue.

3. The on-site data transmission method according to claim 2, wherein: Feeding back the environmental information to the first venue, The sound related to the environmental information is provided to users of the first site.

4. The on-site data transmission method according to claim 3, wherein: The environmental information includes information corresponding to the user's response. A sound corresponding to the response is provided to the user of the first site.

5. The on-site data transmission method according to any one of claims 2 to 4, wherein: The environmental information includes sound picked up by a microphone installed in the second venue.

6. The on-site data transmission method according to any one of claims 2 to 4, wherein: The environmental information includes pre-made sounds.

7. The on-site data transmission method according to claim 6, wherein: The pre-made sounds are different for each venue.

8. The on-site data transmission method according to any one of claims 2 to 4, wherein: The environmental information includes information related to the attribute of the user corresponding to the second sound source, The rendering includes providing processing of the sound based on the attributes.

9. The on-site data transmission method according to any one of claims 1 to 4, wherein: The second sound source information includes position information of the second sound source. The reproduction includes a process of providing the sound of the second sound source subjected to a localization process based on the position information of the second sound source.

10. The on-site data transmission method according to any one of claims 1 to 4, wherein: The second sound source includes a plurality of sound sources.

11. A field data transmission system, comprising: A site data transmission device that transmits, as transmission data, first sound source information related to a sound of a first sound source occurring at a first location of a first venue and position information of the first sound source, and second sound source information related to a second sound source including an ambient sound occurring at a second location of the first venue; and a live data playback device that reproduces the transmission data and provides the sound of the first sound source that has been subjected to positioning processing based on the position information of the first sound source and the sound of the second sound source to the second venue, in, The transmission data includes echo information of the space of the first site, and the reproduction includes processing of providing sound related to the echo of the space to the second site. The sound related to the echo in the space includes a first echo sound corresponding to the sound of the first sound source and a second echo sound corresponding to the sound of the second sound source. The spatial echo information includes first echo information that changes according to a position of the first sound source and second echo information that changes according to a position of the second sound source, and the reproduction includes a process of generating the first echo sound based on the first echo information and a process of generating the second echo sound based on the second echo information.

12. The field data transmission system according to claim 11, wherein: The on-site data playback device transmits environmental information related to the environmental sound of the second venue to outside the second venue.

13. The field data transmission system according to claim 12, wherein: The on-site data playback device feeds back the environmental information to the first venue, The on-site data transmission device provides the sound related to the environmental information to the users of the first site.

14. The field data transmission system according to claim 13, wherein: The environmental information includes information corresponding to the user's response. The on-site data transmission device provides a sound corresponding to the response to the user of the first site.

15. The field data transmission system according to any one of claims 12 to 14, wherein: The environmental information includes sound picked up by a microphone installed in the second venue.

16. The field data transmission system according to any one of claims 12 to 14, wherein: The environmental information includes pre-made sounds.

17. The field data transmission system according to claim 16, wherein: The pre-made sounds are different for each venue.

18. The field data transmission system according to any one of claims 12 to 14, wherein: The environmental information includes information related to the attribute of the user corresponding to the second sound source, The rendering includes providing processing of the sound based on the attributes.

19. The field data transmission system according to any one of claims 11 to 14, wherein: The second sound source information includes position information of the second sound source. The reproduction includes a process of providing the sound of the second sound source subjected to a localization process based on the position information of the second sound source.

20. The field data transmission system according to any one of claims 11 to 14, wherein: The second sound source includes a plurality of sound sources.

21. A field data transmission device, wherein: transmitting, as transmission data, first sound source information related to a sound of a first sound source occurring at a first location of a first venue and position information of the first sound source, and second sound source information related to a second sound source including ambient sound occurring at a second location of the first venue, causing a live data playback device to reproduce the transmission data, and providing the sound of the first sound source subjected to positioning processing based on the position information of the first sound source and the sound of the second sound source to the second venue, wherein the transmission data includes echo information of the space of the first venue, and the reproduction includes processing of providing sound related to the echo of the space to the second venue, The sound related to the echo in the space includes a first echo sound corresponding to the sound of the first sound source and a second echo sound corresponding to the sound of the second sound source. The spatial echo information includes first echo information that changes according to a position of the first sound source and second echo information that changes according to a position of the second sound source, and the reproduction includes a process of generating the first echo sound based on the first echo information and a process of generating the second echo sound based on the second echo information.

22. A live data playback device, wherein: receiving transmission data from a field data transmission device, the field data transmission device transmitting, as the transmission data, first sound source information related to a sound of a first sound source occurring at a first location of a first venue and position information of the first sound source, and second sound source information related to a second sound source including an ambient sound occurring at a second location of the first venue, The on-site data playback device reproduces the transmission data and provides the sound of the first sound source and the sound of the second sound source to the second venue after the positioning process based on the position information of the first sound source is performed. wherein the transmission data includes echo information of the space of the first venue, and the reproduction includes processing of providing sound related to the echo of the space to the second venue, The sound related to the echo in the space includes a first echo sound corresponding to the sound of the first sound source and a second echo sound corresponding to the sound of the second sound source. The spatial echo information includes first echo information that changes according to a position of the first sound source and second echo information that changes according to a position of the second sound source, and the reproduction includes a process of generating the first echo sound based on the first echo information and a process of generating the second echo sound based on the second echo information.

23. A method for transmitting data on site, wherein: transmitting, as transmission data, first sound source information related to a sound of a first sound source occurring at a first location of a first venue and position information of the first sound source, and second sound source information related to a second sound source including ambient sound occurring at a second location of the first venue, causing a live data playback device to reproduce the transmission data, and providing the sound of the first sound source subjected to positioning processing based on the position information of the first sound source and the sound of the second sound source to the second venue, wherein the transmission data includes echo information of the space of the first venue, and the reproduction includes processing of providing sound related to the echo of the space to the second venue, The sound related to the echo in the space includes a first echo sound corresponding to the sound of the first sound source and a second echo sound corresponding to the sound of the second sound source. The spatial echo information includes first echo information that changes according to a position of the first sound source and second echo information that changes according to a position of the second sound source, and the reproduction includes a process of generating the first echo sound based on the first echo information and a process of generating the second echo sound based on the second echo information.

24. A method for playing live data, wherein: receiving transmission data from a field data transmission device, the field data transmission device transmitting, as the transmission data, first sound source information related to a sound of a first sound source occurring at a first location of a first venue and position information of the first sound source, and second sound source information related to a second sound source including an ambient sound occurring at a second location of the first venue, The transmission data is reproduced, and the sound of the first sound source and the sound of the second sound source that have been subjected to positioning processing based on the position information of the first sound source are provided to the second venue, wherein the transmission data includes echo information of the space of the first venue, and the reproduction includes processing of providing sound related to the echo of the space to the second venue, The sound related to the echo in the space includes a first echo sound corresponding to the sound of the first sound source and a second echo sound corresponding to the sound of the second sound source. The spatial echo information includes first echo information that changes according to a position of the first sound source and second echo information that changes according to a position of the second sound source, and the reproduction includes a process of generating the first echo sound based on the first echo information and a process of generating the second echo sound based on the second echo information.

Images