Sound localization method and sound localization system

Abstract

This invention provides a sound image localization method and system that allows monitoring of the localization of the first venue even in a second venue which has a different playback environment from the first venue. [Solution] The sound image localization method is as follows: In the first venue (main venue), a sound image localization device 31 acquires a first sound signal of a first sound source and first localization information which is information regarding the localization of the first sound source 11. In the first venue, based on the first localization information, it determines the first sound processing parameters in the playback environment of the first venue and performs first sound image localization processing on the first sound signal S12. In the second venue (monitor room), a sound image localization device 21 acquires a first sound signal and first localization information S21. Based on the first localization information, it determines the second sound processing parameters in the playback environment of the second venue and performs second sound image localization processing on the first sound signal S22.

Description

Technical Field

[0001] One embodiment of this invention relates to an audio-visual localization method and an audio-visual localization system.

Background Art

[0002] Patent Document 1 discloses a virtual audio-visual localization processing device that changes audio-visual localization information indicating the reproduced audio-visual localization position of a sound source with respect to the listening position of a listener, and performs a process of changing the audio-visual localization position of the sound source based on the changed audio-visual localization information on the acoustic signal of the sound source to obtain a reproduction output of at least two channels.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Patent Document 1 does not disclose a configuration in which separate audio-visual localizations are performed in the main venue (the first venue) and the monitor room (the second venue). Since the reproduction environments are different between the first venue and the second venue, if the localization process is performed using the sound processing parameters of the first venue, different localizations will result between the first venue and the second venue.

[0005] Therefore, an object of one embodiment of this invention is to provide an audio-visual localization method capable of monitoring the localization in the first venue even in a second venue having a reproduction environment different from that of the first venue.

Means for Solving the Problems

[0006] The sound image localization method involves acquiring a first sound signal from a first sound source and first localization information, which is information regarding the localization of the first sound source, in a first venue; determining first sound processing parameters in the playback environment of the first venue based on the first localization information in the first venue; and performing first sound image localization processing on the first sound signal. In a second venue, the first sound signal and the first localization information are acquired; determining second sound processing parameters in the playback environment of the second venue based on the first localization information; and performing second sound image localization processing on the first sound signal. [Effects of the Invention]

[0007] The sound signal processing method allows monitoring of the localization of the sound in the first venue even in the second venue, which has a different playback environment from the first venue. [Brief explanation of the drawing]

[0008] [Figure 1] This is a schematic floor plan showing the main venue (venue 1) 60 and the monitor room (venue 2) 70. [Figure 2] This is a block diagram showing the configuration of the sound localization system 1. [Figure 3] This is a block diagram showing the configuration of the sound image localization device 31. [Figure 4] This flowchart shows the operation of the sound localization method performed by the sound localization device 21 and the sound localization device 31. [Figure 5] This is a schematic floor plan showing the main venue (venue 1) 60 and the monitor room (venue 2) 70 according to modified example 1. [Figure 6] This flowchart shows the operation of the sound localization method performed by the sound localization device 21 and sound localization device 31 according to Modification 1. [Modes for carrying out the invention]

[0009] Figure 1 is a schematic plan view showing an example of the first venue, the main venue 60, and an example of the second venue, the monitor room 70. Figure 2 is a block diagram showing the configuration of the sound image localization system 1. The sound image localization system 1 includes speakers 51A to 51J and a sound image localization device 31 in the main venue 60. The sound image localization system 1 also includes speakers 71A to 71D and a sound image localization device 21 in the monitor room 70.

[0010] The main venue 60 and the monitor room 70 are, for example, separate rooms within the same building. However, in this invention, the main venue 60 and the monitor room 70 do not need to be in the same building. The main venue 60 is a long rectangle in the front-to-back direction when viewed from above, and approximates a cuboid shape in three dimensions. The monitor room 70 is a square when viewed from above, and approximates a cube shape in three dimensions. Of course, the actual main venue 60 is not limited to the cuboid shape shown in Figure 1, and is often a more complex shape. Similarly, the monitor room 70 is not actually a perfect cube, but is often a simpler shape than the main venue 60.

[0011] As shown in Figure 1, speakers 51A to 51J are installed on the walls of the main venue 60 so as to surround the audience seating area of ​​the main venue 60. For example, the front speaker 51F and the rear speaker 51E are installed near the ceiling, while speakers 51A to 51D and speakers 51G to 51J are installed at a height close to the head level of the audience. Speakers 51A to 51J are connected to the sound image localization device 31.

[0012] Speakers 71A to 71D were positioned in the four corners of the monitor room 70. Speakers 71A to 71D were connected to the sound image localization device 21.

[0013] Figure 3 is a block diagram showing the configuration of the sound image localization device 31. Since the sound image localization device 21 and the sound image localization device 31 have the same configuration, Figure 3 shows the configuration of the sound image localization device 31 as a representative example.

[0014] The audio-visual localization device 31 is realized by an acoustic device such as a PC (personal computer), a smartphone, a set-top box, an audio mixer, an audio receiver, or dedicated hardware for signal processing.

[0015] The audio-visual localization device 31 includes a communication unit 11, a processor 12, a RAM 13, a flash memory 14, a display 15, and a user interface (I / F) 16.

[0016] The communication unit 11 has a wireless communication function such as Bluetooth (registered trademark) or Wi-Fi (registered trademark), or a wired communication function such as USB or LAN. The communication unit 11 is connected to speakers 51A to 51J. The audio signal is output to these speakers via the communication unit 11. Also, the communication unit 11 is connected to the audio-visual localization device 21. The audio-visual localization device 31 and the audio-visual localization device 21 transmit and receive various information via their respective communication units 11.

[0017] The display 15 is composed of an LCD, an OLED, etc. The display 15 displays the video output by the processor 12.

[0018] The user I / F 16 is an example of an operation unit. The user I / F 16 is composed of a mouse, a keyboard, or a touch panel, etc. The user I / F 16 receives the operations of the user. Note that the touch panel may be laminated on the display 15.

[0019] The processor 12 is composed of a CPU, a DSP, or a SoC (System on a Chip), etc. The processor 12 reads a program from the flash memory 14 which is a storage medium, and temporarily stores it in the RAM 13, thereby performing various operations. Note that the program does not necessarily need to be stored in the flash memory 14. The processor 12 may, for example, download it from another device such as a server when necessary and temporarily store it in the RAM 13.

[0020] FIG. 4 is a flowchart showing the operation of the audio-visual localization method performed by the audio-visual localization device 21 and the audio-visual localization device 31. First, the audio-visual localization device 31 acquires a first audio signal of a first sound source and first localization information which is information regarding the localization of the first sound source (S11). The first audio signal and the first localization information are, for example, stored in the flash memory 14 in advance. The processor 12 reads out the first audio signal and the first localization information stored in the flash memory 14. Alternatively, the first audio signal and the first localization information may be acquired from another device such as a server via the communication unit 11.

[0021] The first localization information is two-dimensional or three-dimensional logical position information (coordinates represented by numerical values from 0 to 1 with a certain position as the origin) with a predetermined position as the origin. In the main venue 60, the processor 12 obtains first audio processing parameters in the reproduction environment of the first venue based on the acquired first localization information, and performs first audio-visual localization processing on the first audio signal (S12). More specifically, the processor 12 converts the logical coordinates of the first localization information into first physical position information indicating the physical position of the main venue 60 by means of a method such as affine transformation. Affine transformation is an example of geometric transformation. Affine transformation represents the physical coordinates after transformation as a function of the logical coordinates before transformation, and is a method for obtaining the coefficients of the function by means of the least squares method or the like.

[0022] The processor 12 acquires the physical position information of the speakers 51A to 51J in the main venue 60. As the first audio-visual localization processing, the processor 12 performs object-based processing for rendering the first audio signal into audio signals for each speaker.

[0023] Object-based processing is a process that localizes the sound image of a sound source using a panning method such as VBAP (Vector Based Amplitude Panning). The processor 12 calculates the level of the first sound signal to output to speakers 51A to 51J according to the distance between the first sound source and each speaker 51A to 51J, so that the sound image of the first sound source is localized at the position indicated by the first physical position information. The listener perceives localization in the direction of the speaker that outputs a higher level of sound. Therefore, the level of the sound signal output to the speaker closest to the position of the first sound source will be the highest, and the level of the sound signal will be lower for speakers that are farther from the first sound source. As a result, the listener perceives the sound image of the first sound source at the position of the first sound source.

[0024] Meanwhile, the sound localization device 21 acquires the first sound signal and first localization information from the first sound source (S21). The first sound signal and first localization information are stored in the flash memory 14 of the sound localization device 21 beforehand, for example. The processor 12 of the sound localization device 21 reads the first sound signal and first localization information stored in the flash memory 14. Alternatively, the first sound signal and first localization information may be acquired from the sound localization device 31 or other devices such as a server via the communication unit 11 of the sound localization device 21.

[0025] The processor 12 of the sound image localization device 21 determines the second sound processing parameters in the playback environment of the monitor room 70 based on the first localization information, and performs the second sound image localization processing on the first sound signal (S22).

[0026] The processor 12 of the sound localization device 21 performs second sound localization processing in the monitor room 70 using either object-based processing or channel-based processing. Channel-based processing is the process of distributing the first sound signal to multiple channels (in this embodiment, four channels: FL channel, FR channel, SL channel, and SR channel) at a predetermined level ratio based on logical position information. For example, if the logical position information is (X, Y) = (0.5, 1.0) and the position of the first sound source is at the center of the front left and right, the processor 12 of the sound localization device 21 distributes the signal to the FL channel and FR channel at the same level. As a result, the user obtains a sense of localization of the sound source at the center of the front left and right.

[0027] Channel-based processing distributes the first sound signal based on logical position information without considering the physical position of the speakers, thus enabling localization with a lower computational load compared to object-based processing. In other words, the processor 12 of the sound image localization device 21 can achieve a sense of localization close to that of the main venue 60 with a lower computational load than the processor 12 of the sound image localization device 31. Therefore, the processor 12 of the sound image localization device 21 can achieve appropriate sound image localization processing even if it is a general-purpose CPU rather than a DSP dedicated to signal processing.

[0028] However, the processor 12 of the sound localization device 21 may perform the second sound localization processing using object-based processing. Object-based processing converts logical position information into second physical position information indicating the physical position of the monitor room 70, for example, by a method such as affine transformation. The processor 12 of the sound localization device 21 also acquires the physical position information of speakers 71A to 71D in the monitor room 70. As the second sound localization processing, the processor 12 renders the first sound signal into sound signals for each speaker. In this case, although the processor 12 of the sound localization device 21 has a high computational load, it can achieve localization closer to the main venue 60.

[0029] Furthermore, if the shape of the main venue 60 is close to a cube and is a simple shape, the processor 12 of the sound image localization device 31 may perform the first localization processing by channel-based processing.

[0030] As described above, in the sound localization method of this embodiment, the processor 12 of the sound localization device 21 in the monitor room 70 does not refer to the first sound processing parameters of the sound localization device 31 in the main venue 60, but calculates the second sound processing parameters based on the logical position information before rendering. As a result, even in the monitor room 70, which has a different playback environment from the main venue 60, users of the monitor room 70 can monitor the localization of the main venue 60. In other words, the sound localization method of this embodiment makes it possible to realize a new customer experience that was not possible in the past, in which the localization of the main venue 60 can be monitored even in the monitor room 70, which is a different space from the main venue 60. Users of the monitor room 70 may adjust the sound localization position of the first sound source using, for example, a schematic diagram of a cube-shaped logical space displayed on the display 15 while monitoring the localization of the main venue 60.

[0031] (Variation 1) Figure 5 is a schematic plan view showing the main venue 60 and monitor room 70 according to Modification 1.

[0032] In the first modified example, the second sound source 65 is located in front of the main venue 60. The sound from the second sound source 65 is acquired, for example, by a microphone. The sound localization device 31 is connected to the microphone and receives the second sound signal from the second sound source 65. If the second sound source 65 is a device that outputs sound signals, such as an electronic musical instrument, the sound localization device 31 acquires the second sound signal from the second sound source 65 via an audio cable.

[0033] Figure 6 is a flowchart showing the operation of the sound localization method performed by the sound localization device 21 and sound localization device 31 according to Modification 1. The sound localization device 31 acquires the second sound signal of the second sound source 65 and second localization information indicating the physical position of the second sound source 65 in the main venue 60 (S31).

[0034] The second localization information, like the first physical position information, is information indicating two-dimensional or three-dimensional physical coordinates based on a location in the main venue 60. The physical position of the second sound source 65 can be obtained, for example, by beacons and tags that transmit and receive radio waves such as Bluetooth®. At least three beacons are pre-installed in the main venue 60. The second sound source 65 is equipped with a tag. Each beacon transmits and receives radio waves to the tag. Each beacon measures the distance to the tag based on the time difference between transmitting and receiving radio waves. The position of the tag can be uniquely determined by the distance from at least three beacons to the tag and the position information of each beacon. The processor 12 of the sound image localization device 31 obtains the second localization information, which is the physical position of the second sound source 65, by obtaining the position information of the tag measured in this way.

[0035] The processor 12 of the sound localization device 31 determines the first sound processing parameters based on the second localization information and performs the first sound localization processing on the second sound signal (S32). As the first sound localization processing, the processor 12 performs object-based processing to render the second sound signal into sound signals for each speaker. The sound localization device 31 also performs the processing shown in S11 and S12 in Figure 4. That is, in the main venue 60, the processor 12 of the sound localization device 31 determines the first sound processing parameters based on the first physical position information and the second localization information (second physical position information) and performs the first sound localization processing on the first sound signal and the second sound signal, respectively.

[0036] Alternatively, if the shape of the main venue 60 is close to a cube and is a simple shape, the processor 12 of the sound image localization device 31 may perform the first localization processing by channel-based processing.

[0037] Meanwhile, the sound localization device 21 acquires the second sound signal and second localization information from the second sound source (S41). The second sound signal and second localization information are acquired from the sound localization device 31 via the communication unit 11 of the sound localization device 21.

[0038] The processor 12 of the sound image localization device 21 converts the second localization information, which is physical position information, into logical position information indicating a logical position using a method such as affine transformation, and then determines the second sound processing parameters based on the converted logical position information and performs the second sound image localization processing on the second sound signal (S42). The processor 12 of the sound image localization device 21 performs the second sound image localization processing using channel-based processing according to the playback environment in the monitor room 70. The sound image localization device 21 also performs the processing shown in S21 and S22 in Figure 4. That is, the sound image localization device 21 determines the second sound processing parameters using channel-based processing based on the converted logical position information (first logical position information) and the logical position information contained in the first localization information (second logical position information) and performs the second sound image localization processing on the first sound signal and the second sound signal, respectively.

[0039] Thus, the processor 12 of the sound localization device 21 converts the physical position information of the main venue 60 into logical position information, specifically first logical position information. To achieve this, it distributes the second sound signal using channel-based processing without considering the physical position information of speakers 71A to 71D in the monitoring room 70. Therefore, the processor 12 of the sound localization device 21 can monitor a sense of localization similar to that of the second sound source 65 in the main venue 60, while having a lower computational load compared to object-based processing. This enables a new customer experience that was previously impossible.

[0040] However, the processor 12 of the sound localization device 21 may perform the second sound localization processing using object-based processing. In this case, the processor 12 of the sound localization device 21 converts the second localization information into first logical position information, and then further converts the first logical position information into third physical position information indicating the physical position of the monitor room 70, for example, by a method such as affine transformation. The processor 12 of the sound localization device 21 then acquires the physical position information of speakers 71A to 71D in the monitor room 70. As the second sound localization processing, the processor 12 renders the first sound signal and the second sound signal into sound signals for each speaker. In this case, although the processor 12 of the sound localization device 21 has a high computational load, it can achieve localization closer to the main venue 60.

[0041] (Other examples) The first and second sound localization processes may include processes for generating indirect sound, not just direct sound. The indirect sound generation process is a process that generates an early reflection control signal that reproduces the early reflection sound and a reverberation control signal that reproduces the reverberation sound, respectively.

[0042] The processor 12 of the sound localization device 31 generates early reflection control signals and reverberation control signals for the first sound signal by, for example, convolving the impulse response data of the main venue 60 into the first sound signal. The processor 12 outputs the early reflection control signals and reverberation control signals in addition to the direct sound from multiple speakers. As a result, the audience hears the early reflection control signals and reverberation control signals in addition to the direct sound. The processor 12 of the sound localization device 31 may also generate early reflection control signals and reverberation control signals for the second sound signal by, for example, convolving the impulse response data of the main venue 60 into the second sound signal.

[0043] Impulse response data is measured, for example, by emitting a test sound (pulse sound) at a predetermined location in the main venue 60 and capturing the sound with a measurement microphone (not shown). Alternatively, the impulse response may be obtained by simulation based on, for example, the sound ray method or the virtual image method. The sound ray method is a technique that tracks the trajectory (sound ray) of sound radiated from a sound source and calculates the time pattern of the energy of the sound ray passing through the listening position. A simulation using the sound ray method determines the direction, arrival time, and arrival level from each virtual sound source at the listening position, based on the energy of the sound ray in the listening area, assuming that each sound ray is a virtual sound image of the reverberation. The virtual image method is a technique that creates a virtual image (virtual sound source) of the sound source against the wall surface of the space as a virtual sound source, and determines the direction, arrival time, and arrival level from each virtual sound source at the listening position. The processor 12 may generate impulse responses representing the direction, arrival time, and arrival level of each virtual sound source determined by simulation, and perform processing to localize indirect sound by convolving these impulse responses into the sound source signal. The impulse response data may be stored in the flash memory 14. Alternatively, the impulse response data may be downloaded from a server (not shown) or the like each time.

[0044] Similarly, the processor 12 of the sound localization device 21 generates the initial reflection control signal and reverberation control signal for the first sound signal by convolving the impulse response data of the main venue 60 into the first sound signal. Users in the monitoring room 70 can monitor the initial reflection control signal and reverberation control signal in addition to the localization of the direct sound of the first sound source.

[0045] Furthermore, the processor 12 of the sound localization device 21 may generate an early reflection control signal and a reverberation control signal for the second sound signal by convolving the impulse response data of the main venue 60 into the second sound signal. Users of the monitoring room 70 can monitor the early reflection control signal and the reverberation control signal in addition to the localization of the direct sound of the second sound source.

[0046] The description of this embodiment is illustrative in all respects and not restrictive. The scope of the invention is indicated by the claims, rather than by the embodiments described above. Furthermore, the scope of the invention is intended to include all modifications within the meaning and scope equivalent to the claims. [Explanation of symbols]

[0047] 1: Sound localization system, 11: Communication unit, 12: Processor, 13: RAM, 14: Flash memory, 15: Display unit, 16: User interface, 21: Sound localization device, 31: Sound localization device, 51A~51J: Speakers, 71A~71D: Speakers, 60: Main venue, 65: Second sound source, 70: Monitor room

Claims

1. At the first venue, the first sound signal of the first sound source and the first localization information, which is information regarding the localization of the first sound source, are acquired. In the first venue, based on the first localization information, first sound processing parameters in the playback environment of the first venue are determined, and first sound image localization processing is performed on the first sound signal. In the second venue, the first sound signal and the first localization information are acquired, and based on the first localization information, the second sound processing parameters in the playback environment of the second venue are determined, and the first sound signal is subjected to second sound image localization processing. Sound image localization method.

2. In the first venue, the second sound signal of the second sound source and second localization information indicating the physical position of the second sound source within the first venue are acquired. In the first venue, the first sound processing parameters are determined based on the second localization information, and the first sound image localization processing is performed on the second sound signal. In the second venue, the second localization information is converted into first logical position information indicating a logical position, the second sound processing parameters are determined based on the first logical position information, and the second sound signal is subjected to the second sound image localization processing. The sound image localization method according to claim 1.

3. The first localization information includes second logical position information indicating the logical position of the first sound source, In the first venue, the second logical position information is converted into first physical position information indicating a physical location within the first venue. At the first venue, the first sound processing parameters are determined based on the first physical position information and the second localization information, and the first sound image localization processing is performed on the first sound signal and the second sound signal, respectively. In the second venue, the second sound processing parameters are determined based on the first logical position information and the second logical position information, and the second sound image localization processing is performed on the first sound signal and the second sound signal, respectively. The sound image localization method according to claim 2.

4. The first sound localization process includes object-based processing that renders the first sound signal into sound signals for each speaker based on the first physical position information and the physical position information for each speaker installed in the first venue. The sound image localization method according to claim 3.

5. The second sound image localization process includes channel-based processing that generates sound signals for each channel based on the first logical position information and the second logical position information. The sound image localization method according to claim 3.

6. In the second venue, the second logical position information is converted into second physical position information indicating a physical location within the second venue. The second sound localization process includes object-based processing that renders the first sound signal into sound signals for each speaker based on the second physical position information and the physical position information for each speaker installed in the second venue. The sound image localization method according to claim 3.

7. In the second venue, the first logical position information is converted into a third physical position information indicating the physical location within the second venue. The second sound localization process includes object-based processing that renders the second sound signal into sound signals for each speaker based on the third physical position information and the physical position information for each speaker installed in the second venue. The sound image localization method according to claim 6.

8. The second sound source is acquired via a microphone installed in the first venue. The sound image localization method according to claim 2 or claim 3.

9. The first sound localization device in the first venue acquires the first sound signal of the first sound source and first localization information, which is information regarding the localization of the first sound source. Based on the first localization information, first sound processing parameters in the playback environment of the first venue are determined, and first sound image localization processing is performed on the first sound signal. The second sound localization device in the second venue acquires the first sound signal and the first localization information, determines the second sound processing parameters in the playback environment of the second venue based on the first localization information, and performs the second sound localization processing on the first sound signal. Sound localization system.

Images