Acoustic signal reproducing device, method, and program

Abstract

This acoustic signal reproducing device includes: a content reproduction unit 101 for generating an acoustic signal of content to be reproduced; and an acoustic control unit 102 for generating, on the basis of the acoustic signal, an acoustic signal for a speaker and an acoustic signal for an earphone different from the acoustic signal for the speaker, outputting sound based on the acoustic signal for the speaker from a speaker 103 installed in a venue, and outputting sound based on the acoustic signal for the earphone from an earphone 104 which is a wearable device for reproducing sound without completely shielding an external ear hole and which is worn by audience present in the venue.

Description

Acoustic signal reproduction device, method, and program

[0001] The disclosed technology relates to a technology for reproducing an acoustic signal.

[0002] In conventional theater, movie, and live music events, the audience listens to the sound from PA speakers installed in the venue, allowing a large number of people to share the sound (see, for example, Non-Patent Document 1).

[0003] Additionally, for movies and live music performances, multiple PA speakers are installed in the venue to create a surround sound effect.

[0004] PA INFORMATION For Beginner, "What is a PA system that delivers sound to the audience?" [online] [Retrieved June 1, 2023], Internet <URL: https: / / pabasic.com / engineer / pakikinokiso / pa-system / >

[0005] However, while it is possible to reproduce three-dimensional sound by physically placing speakers to the left, right, or behind the user, PA speakers cannot produce sounds with an elevation angle or sound images that are closer to the user than speakers.

[0006] In addition, it is difficult for PA speakers to reproduce not only stereophonic sound but also ASMR, which creates the illusion that sounds are coming from close to the user's ears.

[0007] As such, there are sound effects that cannot be achieved by playback through speakers alone.

[0008] The disclosed technology aims to provide an audio signal reproducing device, method, and program that make it possible to obtain sound effects that cannot be achieved with speakers alone.

[0009] One aspect of the disclosed technology is an audio signal playback device that includes a content playback unit that generates an audio signal for content to be played back, and an audio control unit that generates an audio signal for a speaker and an audio signal for an earphone that is different from the audio signal for the speaker based on the audio signal, and outputs the sound based on the audio signal for the speaker from a speaker installed in the venue, and outputs the sound based on the audio signal for the earphone from earphones, which are wearable devices worn by spectators in the venue and play sound without completely blocking the external auditory canal.

[0010] According to the disclosed technology, it is possible to obtain acoustic effects that cannot be achieved with speakers alone.

[0011] Fig. 1 is a diagram showing an example of the functional configuration of an audio signal reproducing device, Fig. 2 is a diagram showing an example of a processing procedure of an audio signal reproducing method, and Fig. 3 is a diagram showing an example of the functional configuration of a computer.

[0012] Hereinafter, embodiments of the disclosed technology will be described with reference to the drawings. Note that components having the same functions in the drawings are given the same reference numerals, and redundant description will be omitted.

[0013] 1, the audio signal reproduction device includes, for example, a content reproduction unit 101 and an audio control unit 102. The audio signal reproduction device may further include at least one of a delay control unit 105, a filter processing unit 106, a speaker 103, an earphone 104, a microphone 107, an amplifier 108, and a transmission unit 109.

[0014] The microphone 107 is installed in the venue indicated by the two-dot chain line in Fig. 1. For example, the microphone 107 is placed in a position where it can pick up the voices of performers performing at the venue and sounds generated by the performance at the venue.

[0015] Examples of venues include movie theaters, live music venues, and kabuki theater venues. Note that the venue may also be a place where a home theater system is installed.

[0016] The sound signal reproducing method is realized, for example, by each component of the sound signal reproducing device performing the processes of steps S101 to S102 shown in FIG.

[0017] <Content Reproducing Unit 101> The content reproducing unit 101 generates an acoustic signal of the content to be reproduced (step S101).

[0018] The generated acoustic signal is output to the acoustic control unit 102 .

[0019] The content playback unit 101 may be a PC, a smartphone, a tablet terminal, a CD player, or a digital audio player that is arranged outside the audio signal playback device. In other words, the audio signal playback device does not need to have the content playback unit 101.

[0020] <Acoustic Control Unit 102> The acoustic signal generated by the content playback unit 101 is input to the acoustic control unit 102.

[0021] First, the acoustic control unit 102 generates an acoustic signal for a speaker and an acoustic signal for an earphone, which is different from the acoustic signal for a speaker, based on the input acoustic signal.

[0022] The generated speaker acoustic signal is output to the speaker 103. As will be described later, the speaker 103 outputs a sound based on the input speaker acoustic signal.

[0023] The generated earphone acoustic signal is output to the earphone 104. As will be described later, the earphone 104 outputs sound based on the input earphone acoustic signal.

[0024] In this way, the acoustic control unit 102 generates an acoustic signal for a speaker and an acoustic signal for an earphone that is different from the acoustic signal for the speaker based on the input acoustic signal, and outputs the sound based on the acoustic signal for the speaker from a speaker 103 installed in the venue, and outputs the sound based on the acoustic signal for the earphone from earphones 104, which are wearable devices worn by spectators in the venue and reproduce sound without completely blocking the external auditory canal (step S102).

[0025] A collected sound signal collected by a microphone 107 installed in the venue may be input to the acoustic control unit 102. There may be a plurality of microphones 107.

[0026] In this case, the audio control unit 102 generates an audio signal for the speaker and an audio signal for the earphone based on the audio signal and the audio signal collected by the microphone 107 installed in the venue. The subsequent processing is the same as described above.

[0027] In this way, the acoustic control unit 102 may have the function of a mixer.

[0028] An example of the speaker audio signal is an audio signal corresponding to at least one track constituting the multi-track when the audio signal generated by the content playback unit 101 is a multi-track signal. In this case, the earphone audio signal is an audio signal corresponding to at least one other track constituting the multi-track.

[0029] Another example of the speaker audio signal is an audio signal generated by the content playback unit 101. In this case, the earphone audio signal is a collected audio signal collected by the microphone 107.

[0030] Another example of the speaker audio signal is an audio signal corresponding to background music. In this case, the earphone audio signal is at least one of an audio signal corresponding to sound effects and an audio signal corresponding to the voice of a performer performing at a venue, the voice being collected by the microphone 107.

[0031] Another example of the speaker audio signal is an audio signal corresponding to a low-pitched sound, whereas the earphone audio signal is an audio signal corresponding to a sound other than a low-pitched sound.

[0032] Furthermore, an audio signal corresponding to ASMR (Autonomous Sensory Meridian Response) content may be included in the earphone audio signal.

[0033] Assume that there are multiple performers in the venue, and microphones 107 are arranged to capture the voices of the multiple performers. Also, assume that each audience member has pre-selected at least one of the multiple performers. In this case, the audio signal of at least one of the multiple performers pre-selected by each audience member may be included in the earphone audio signal corresponding to that audience member.

[0034] In this way, a different acoustic signal for each audience member may be included in the earphone acoustic signal corresponding to each audience member.

[0035] Furthermore, an audio signal for emergency announcements such as instructions for leaving the venue may be included in the earphone audio signal.

[0036] <Speaker 103> The speaker 103 receives the speaker acoustic signal generated by the acoustic control unit 102.

[0037] The speaker 103 outputs a sound based on the input speaker acoustic signal.

[0038] The speaker 103 is a speaker installed in the venue. Note that there may be multiple speakers 103.

[0039] The sound signal reproducing device may include an amplifier 108 that amplifies the signal input to the speaker 103. The amplifier 108 is indicated by a dashed line in FIG.

[0040] In this case, the speaker 103 outputs a sound based on the signal amplified by the amplifier 108 .

[0041] <Earphone 104> The earphone 104 receives an earphone acoustic signal generated by the acoustic control unit 102.

[0042] The earphone 104 outputs a sound based on the input earphone acoustic signal.

[0043] The earphones 104 are wearable devices worn by audience members in the venue that reproduce sound without completely blocking the ear canals. There may be multiple earphones 104. An example of such a wearable device is a so-called open-ear earphone. By using such a wearable device as the earphones 104, audience members can hear sounds other than those output from the earphones 104, in other words, sounds from the venue, allowing the audience members to enjoy a sense of realism.

[0044] The sound signal reproducing device may include at least one of a delay control section 105 and a filter processing section 106, which will be described below. In Fig. 1, the delay control section 105 and the filter processing section 106 are indicated by dashed lines.

[0045] Signals are input to the earphones 104 via a wired or wireless connection. When signals are input via a wired connection, the audio signal reproducing device may include an amplifier (not shown) that amplifies the signals input to the earphones 104. In this case, the earphones 104 output sound based on the signals amplified by the amplifier. For wireless transmission, existing wireless technologies such as 800 MHz band wireless technology and Bluetooth (registered trademark) may be used, or proprietary wireless technologies may be used. The audio signal reproducing device may include a transmitter 109 that transmits signals wirelessly to the earphones 104.

[0046] <Delay Control Unit 105> The delay control unit 105 receives the speaker audio signal and earphone audio signal generated by the audio control unit 102.

[0047] The delay control unit 105 delays at least one of the input audio signal for the speaker and the audio signal for the earphone.

[0048] The delayed speaker acoustic signal is output to the speaker 103. The speaker 103 outputs a sound based on the delayed speaker acoustic signal.

[0049] The delayed earphone acoustic signal is output to the earphone 104. The earphone 104 outputs sound based on the delayed earphone acoustic signal.

[0050] These delays are performed, for example, so that the sound based on the speaker audio signal and the sound based on the earphone audio signal reach the audience at approximately the same time. When the speaker audio signal and the earphone audio signal contain the same signal, synchronizing the sound based on the speaker audio signal and the sound based on the earphone audio signal in this way can prevent double sounds from being heard.

[0051] In addition, if there are multiple spectators, each spectator will be located at a different position in the venue, so the delay control unit 105 may take into account the position of each spectator in the venue when performing processing to ensure that the sound based on the speaker audio signal and the sound based on the earphone audio signal reach the spectators at approximately the same time.

[0052] The delay control unit 105 may adjust at least one of the delay amount of the speaker audio signal and the delay amount of the earphone audio signal in real time, for example, using a signal obtained from a microphone placed at the position of the audience in the venue.

[0053] The delay control unit 105 may also determine in advance at least one of the delay amount of the speaker audio signal and the delay amount of the earphone audio signal based on the positions of the speaker 103, the positions of the audience, and the speed of sound.

[0054] For example, in this manner, the delay control unit 105 delays at least one of the sound based on the speaker audio signal and the sound based on the earphone audio signal (step S105).

[0055] <Filter Processing Unit 106 > The delay control unit 105 receives the speaker acoustic signal and earphone acoustic signal generated by the acoustic control unit 102 .

[0056] The filter processing unit 106 performs filtering on the earphone acoustic signal using at least one of a filter for canceling out at least one of the characteristics due to the housing position of the earphone 104 and the characteristics due to the shape of the audience's ears, a filter that approximates a head-related transfer function (HRTF), and a room impulse response of the room reverberation components, and generates a filtered earphone acoustic signal.

[0057] The generated filtered earphone signal is output to the earphone 104. The earphone 104 outputs sound based on the filtered earphone acoustic signal.

[0058] Generally, when a wearable device that reproduces sound without completely blocking the external ear canal is used, the sound image is often perceived as being located upward. An example of the housing position characteristic of the earphone 104 is a characteristic that causes the sound image to be perceived as being located upward. For example, the housing position characteristic of the earphone 104 that will actually be used at the venue is measured in advance using a dummy head. The filter processing unit 106 performs the above-described filter processing using the measured housing position characteristic of the earphone 104.

[0059] By using a filter that approximates the head-related transfer function, it becomes possible to localize the direction of sound and reproduce a sound with a sense of realism.In addition, by using a filter that approximates the head-related transfer function, it becomes possible to reproduce a sound in a binaural manner, for example.

[0060] When the head-related transfer functions corresponding to the audience are known, the filter processing unit 106 performs filtering using the head-related transfer functions corresponding to the audience.

[0061] When there are multiple audience members and the head-related transfer functions corresponding to each audience member are known, the filter processing unit 106 performs filtering using the head-related transfer functions corresponding to each audience member to generate multiple filtered earphone acoustic signals, which are then output to the corresponding audience member.

[0062] Even when there are multiple spectators, the filter processing unit 106 may perform filtering using one head-related transfer function that is common to the multiple spectators.

[0063] If head tracking information can be obtained from the audience, the filter processing unit 106 may perform filtering using a head-related transfer function, taking the head tracking information into consideration.

[0064] By performing filtering using room impulse responses, the effects of reverberation caused by the size of the venue can be imparted to the sound source being played near the ears, making it possible to present sounds that are occurring within the same room.

[0065] The filter processing unit 106 may also perform filtering so that the sound to be reproduced is localized at a position closer to the audience than the speaker 103 .

[0066] When the acoustic signal reproducing device includes both the delay control unit 105 and the filter processing unit 106, the signal processed by the delay control unit 105 and the filter processing unit 106 is output to the earphone 104. The earphone 104 outputs sound based on the processed signal.

[0067] In this case, it does not matter which process is performed first, the process by the delay control unit 105 or the process by the filter processing unit 106. That is, although the delay control unit 105 is arranged before the filter processing unit 106 in FIG. 1 , the filter processing unit 106 may be arranged before the delay control unit 105.

[0068] The audio signal reproducing device and method may be used in, for example, Kabuki. An example of processing when the audio signal reproducing device and method are used in Kabuki will be described below. In Kabuki, effects are usually produced using traditional Japanese musical instruments and tsuke (a traditional Japanese bell). However, the usual production techniques are used as they are, and open-ear earphones are placed in the audience seats as earphones 104. The earphones 104 reproduce sounds of swords and horses in accordance with the performers' progress. A stereophonic effect can be added to the sounds of swords, horses, and speaking, creating a sense of realism. The performer's breathing and other sounds can be picked up from a pin microphone, which is the performer's microphone 107, and reproduced from close to the performer's ear, creating an effect that makes the performer feel closer to the audience.

[0069] The audio signal playback device and method may be used, for example, at idol concerts. An example of the process when the audio signal playback device and method are used at an idol concert is described below. At idol concerts, multiple vocalists sing, and live singing audio is acquired, mixed appropriately, and played over a PA system, or pre-recorded audio is played over a PA system. By playing the audio of the favorite performer from these mixed audio sources close to the ear, the audio can be emphasized and heard. Furthermore, by changing the audio of the favorite performer for each audience member attending the live performance, the content close to the ear can be changed while maintaining a single live performance. Furthermore, by utilizing nearby HRTFs, it is possible to create the effect of the singer being performed next to, in front of, or close to the wearer's ear, rather than from a very distant location in the venue. In addition to singing, musical performances and individual messages can be sent to users. Multiple variations can be created by changing the combination of performances for each user and performance. By issuing instructions to the earphones, such as exit instructions or emergency announcements, it is possible to clearly communicate who is currently being guided.

[0070] The audio signal reproduction device and method may be used, for example, in movies. An example of the processing when the audio signal reproduction device and method are used in movies will be described below. In this case, existing surround speakers are arranged as the speakers 103, and open-ear earphones are provided as the earphones 104 at each seat. The open-ear earphones reproduce sounds that go beyond conventional acoustic expression, such as reproducing stereophonic sounds and reproducing sounds close to the ears, which is impossible with stereophonic sound reproduced by speakers. To reproduce stereophonic sounds such as the voice of a criminal close to the ear or the sound of a mosquito flying close to the ear, the sound is reproduced by playing back sounds convoluted with a transfer function from the sound source to the ears, known as HRTF, from earphones in both ears. Regarding delay correction, since the seat positions and speaker positions are fixed, the distance from the speakers to the seats is measured, and the delay amount is calculated and corrected from the speed of sound.

[0071] [Modifications] The specific configurations of the embodiments of the disclosed technology are not limited to the configurations described above. The specific configurations of the embodiments of the disclosed technology can be appropriately modified in design, etc., within the scope of the spirit of the embodiments of the disclosed technology.

[0072] The various processes described in the embodiments of the disclosed technology may not only be performed chronologically in the order described, but may also be performed in parallel or individually depending on the processing capacity of the device performing the processes or as needed.

[0073] For example, data may be exchanged directly between the components of the sound signal reproducing device, or may be exchanged via a storage unit (not shown).

[0074] [Program, Recording Medium] The processing of each unit of the above-described audio signal reproducing device may be realized by a computer, in which case the processing content of the functions to be possessed by each device is described by a program. Then, by loading this program into storage unit 1020 of computer 1000 shown in Figure 3 and operating arithmetic processing unit 1010, input unit 1030, output unit 1040, display unit 1060, etc., the various processing functions of each of the above-described devices are realized on the computer.

[0075] The program describing the processing contents can be recorded on a computer-readable recording medium, such as a non-transitory recording medium, specifically a magnetic recording device, an optical disk, or the like.

[0076] The program may be distributed, for example, by selling, transferring, lending, etc. a portable recording medium such as a DVD or CD-ROM on which the program is recorded. Furthermore, the program may be stored in a storage device of a server computer, and then transferred from the server computer to another computer via a network, thereby distributing the program.

[0077] A computer executing such a program, for example, first stores the program recorded on a portable recording medium or transferred from a server computer in its own non-transitory storage device, auxiliary storage unit 1050. Then, when executing a process, the computer loads the program stored in auxiliary storage unit 1050, its own non-transitory storage device, into storage unit 1020 and executes processing in accordance with the loaded program. Alternatively, as another form of execution of this program, the computer may load the program directly from a portable recording medium into storage unit 1020 and execute processing in accordance with the program. Furthermore, the computer may execute processing in accordance with the received program each time a program is transferred from a server computer to the computer. Alternatively, the server computer may not transfer the program to the computer, but may instead execute the processing function by issuing an execution instruction and obtaining the results, thereby executing the above-described processing through a so-called ASP (Application Service Provider) type service. Note that, in this embodiment, the program includes information used for processing by a computer that is equivalent to a program (e.g., data that is not a direct instruction to the computer but has properties that define computer processing).

[0078] In this embodiment, the device is configured by executing a predetermined program on a computer, but at least a part of the processing may be realized by hardware. For example, the content playback unit 101, the audio control unit 102, the delay control unit 105, the filter processing unit 106, and the transmission unit 109 may be configured by processing circuits.

[0079] It goes without saying that other modifications are possible without departing from the spirit of the present invention.

[0080] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. An audio signal reproducing device comprising: a content reproducing unit which generates an audio signal for content to be reproduced; and an audio control unit which generates an audio signal for a speaker and an audio signal for an earphone which is different from the audio signal for the speaker based on the audio signal, outputs the sound based on the audio signal for the speaker from a speaker installed in the venue, and outputs the sound based on the audio signal for the earphone from earphones which are wearable devices worn by spectators in the venue and reproduce sound without completely blocking the external auditory canal.

2. An audio signal reproducing device according to claim 1, wherein the audio control unit generates the audio signal for the speaker and the audio signal for the earphone based on a sound collection signal collected by a microphone installed in the venue and the audio signal.

3. The audio signal reproducing device according to claim 1, further comprising a delay control unit that delays at least one of the sound based on the speaker audio signal and the sound based on the earphone audio signal.

4. An acoustic signal reproducing device according to claim 1, further comprising a filter processing unit that performs filtering on the earphone acoustic signal using at least one of a filter for canceling out at least one of the characteristics of the earphone housing position and the characteristics due to the shape of the audience's ears and a filter that approximates a head-related transfer function, and generates a filtered earphone acoustic signal, wherein the earphone outputs sound based on the filtered earphone acoustic signal.

5. An audio signal reproducing method including: a content reproducing step in which a content reproducing unit generates an audio signal of the content to be reproduced; and an audio control step in which an audio control unit generates an audio signal for a speaker and an audio signal for an earphone different from the audio signal for the speaker based on the audio signal, outputs the sound based on the audio signal for the speaker from a speaker installed in the venue, and outputs the sound based on the audio signal for the earphone from earphones, which are wearable devices worn by spectators in the venue and reproduce sound without completely blocking the external ear canal.

6. A program for causing a computer to function as each part of the audio signal reproducing device according to any one of claims 1 to 4.

Images