Signal transmission method, signal generation method, signal reproduction method, voice signal processing program, voice transmission device, voice reproduction device, and voice transmission reproduction system

Abstract

The present invention provides a signal transmission method capable of suppressing unnatural sound images.　This method is a transmission method of a transmission signal T including voice signals of a plurality of channels. The transmission signal T including a binauralized first category voice signal F and a non-binauralized second category voice signal S is transmitted in real time. The binauralized first category voice signal F is a voice signal of a portion of channels, and the non-binauralized second category voice signal S is a voice signal of another channel. The second category voice signal S is a signal for which unnaturalness is audibly perceived more easily than the first category voice signal F with respect to positional deviation and movement.

Description

Signal transmission method, signal generation method, signal reproduction method, audio signal processing program, audio transmission device, audio reproduction device, and audio transmission and reproduction system

[0001] The present invention relates to a signal transmission method, a signal generation method, a signal reproduction method, an audio signal processing program, an audio transmission device, an audio reproduction device, and an audio transmission and reproduction system for creating a stereophonic audio signal to be reproduced particularly through headphones or the like.

[0002] 2. Description of the Related Art Conventionally, there have been VR headphones and HMDs (Head Mounted Displays) (hereinafter referred to as "HMDs, etc.") that are capable of playing content such as movies, VR (Virtual Reality), and AR (Augmented Reality).

[0003] These HMDs and the like use head-related transfer functions (HRTFs) that take into account the direction from the listener to the sound source to reproduce binaural sound, a three-dimensional sound that localizes sound outside the listener's head, so that a wider sound field can be perceived. By measuring the direction data of the listener's head relative to the sound source using an acceleration sensor, position sensor, etc. (hereinafter referred to as "head tracking"), more accurate binaural sound, a two-channel (2ch) three-dimensional sound, can be generated (hereinafter referred to as "binauralization"), resulting in a more realistic sound image.

[0004] Furthermore, in the past, when binauralizing using HRTF, calculations on actual audio signals often used head-related impulse responses (hereinafter referred to as "HRIRs") that represent head-related transfer functions on the time axis (hereinafter HRTFs or HRIRs will be simply referred to as "HRTFs, etc.").

[0005] Patent Literature 1 describes a technique for capturing, acoustically preprocessing, encoding, decoding, and binaural rendering of directional audio of an audio scene. The technique of Patent Literature 1 includes an apparatus adapted to modify directional characteristics of the captured directional audio in response to spatial data of a microphone system capturing the directional audio. The apparatus is configured to modify directional characteristics of the received directional audio by head tracking in response to the received spatial data.

[0006] Japanese Patent Application Laid-Open No. 2021-5822

[0007] In conventional head tracking devices such as that described in Patent Document 1, directional information about the listener's head is transmitted, and the device that receives this information updates the HRTF, etc., which causes a delay in the update equivalent to the round-trip communication time, resulting in an unnatural sound image.

[0008] The present invention has been made in view of the above circumstances, and aims to solve the above-mentioned problems.

[0009] A signal transmission method of the present invention is a method for transmitting signals including audio signals of multiple channels, characterized in that the signals including binauralized first category audio signals and non-binauralized second category audio signals are transmitted in real time. The binauralized first category audio signals are audio signals of some channels, and the non-binauralized second category audio signals are audio signals of other channels. The second category audio signals are signals that are more likely to be perceived as auditorily unnatural when displaced or moved than the first category audio signals. The signal transmission method of the present invention is characterized in that the ease with which the auditory unnaturalness of the audio signals of the multiple channels is perceived is determined based on azimuth and elevation angle. The first category audio signals are low-frequency components, and the second category audio signals are high-frequency components. The first category audio signals are representative point signals obtained by grouping the signals including the audio signals of the multiple channels in a direction that is smaller than the number of channels of the signals. A signal generation method of the present invention is a signal generation method for generating a signal including audio signals of multiple channels, which binaurally processes only first category audio signals based on directional information of a received listener, and generates the signal including the binaurally processed first category audio signals and second category audio signals that are not binaurally processed.A signal reproduction method of the present invention is a signal reproduction method for generating a signal including audio signals of multiple channels, which acquires a signal transmitted by the signal transmission method or a signal generated by the signal generation method, binaurally processes the received second category audio signals, and generates a binaural signal by adding the binaurally processed second category audio signals and the first category audio signals that have already been binaurally processed when received.The signal reproduction method of the present invention is a signal reproduction method executed by an audio generation and reproduction system including an audio transmission device that generates and transmits a signal including audio signals of multiple channels, and an audio reproduction device that reproduces the signal transmitted from the audio transmission device, characterized in that the audio transmission device generates a representative point signal that groups the signal including the audio signals of multiple channels in directions fewer than the number of channels of the signal, transmits the representative point signal in real time, and binauralizes the representative point signal to generate a binaural signal.The audio signal processing program of the present invention is an audio signal processing program executed by the audio transmission device, characterized in that the audio transmission device binauralizes only first category audio signals based on received listener direction information, and generates a signal including the binauralized first category audio signals and second category audio signals that are not binauralized. The audio signal processing program of the present invention is an audio signal processing program executed by an audio playback device, causing the audio playback device to acquire a signal transmitted by the signal transmission method or a signal generated by the signal generation method, binauralize the received second category audio signal, and generate a binaural signal by combining the binauralized second category audio signal with the first category audio signal that has already been binauralized when received.The audio transmission device of the present invention generates a signal including audio signals of multiple channels, and is characterized by comprising: a first binauralization unit that binauralizes only the first category audio signal based on direction information of the listener received from the audio playback device, and a signal generation unit that generates the signal including the first category audio signal binauralized by the first binauralization unit and the second category audio signal that has not been binauralized.The audio playback device of the present invention is a playback device that plays back a signal including audio signals of multiple channels, and is characterized by comprising: a signal acquisition unit that acquires the signal including a binauralized first category audio signal and a non-binauralized second category audio signal; a second binauralization unit that binauralizes the second category audio signal acquired by the signal acquisition unit; a binaural generation unit that combines the second category audio signal binauralized by the second binauralization unit with the first category audio signal to generate a binaural signal; and an audio output unit that outputs the binaural signal generated by the binaural generation unit. The audio transmission and playback system of the present invention is an audio generation and playback system including an audio transmission device that generates a signal including audio signals of a plurality of channels, and an audio playback device that plays back the signal generated and transmitted by the audio transmission device, wherein the audio transmission device includes a first binauralization unit that binauralizes only a first category audio signal based on direction information of a listener received from the audio playback device, a signal generation unit that generates the signal including the first category audio signal binauralized by the first binauralization unit and a second category audio signal that is not binauralized, and a second category audio signal that is not binauralized. and a transmission unit that transmits the signal generated by the audio transmission device in real time, and the audio playback device is characterized by comprising a signal acquisition unit that acquires the signal generated by the audio transmission device, a second binauralization unit that binauralizes the second category audio signal acquired by the signal acquisition unit, a binaural generation unit that generates a binaural signal by combining the second category audio signal binauralized by the second binauralization unit and the first category audio signal, an audio output unit that outputs the binaural signal generated by the binaural generation unit, and a direction transmission unit that transmits direction information of the listener to the audio transmission device.

[0010] According to the present invention, by transmitting a transmission signal in real time that includes a binauralized first category audio signal and a non-binauralized second category audio signal, it is possible to share the binauralization between the transmitting side and the receiving side, thereby providing a signal transmission method that can prevent the sound image from becoming unnatural.

[0011] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments.

[0012] <Embodiment> [Control Configuration of Audio Transmission and Reproduction System X] First, the control configuration of an audio transmission and reproduction system X according to an embodiment of the present invention will be described with reference to Fig. 1. The audio transmission and reproduction system X includes an audio transmission device 1 and an audio reproduction device 2.

[0013] The audio transmission device 1 is a device that encodes and transmits audio signals of multiple channels (hereinafter referred to as "source audio signals"). In addition, the audio transmission device 1 may be capable of generating the source audio signal itself by using or synthesizing the original audio signal. In this embodiment, the audio transmission device 1 is, for example, a parent terminal such as a smartphone, a mobile phone, or an HMD, a gaming personal computer (PC) that may be portable, a consumer or arcade game machine, a server, a television, a television (video) conferencing system, remote conferencing equipment, an audio hearing aid, a hearing aid, other home appliances, equipment at movie theaters or public viewing venues, a dedicated audio encoder, a switcher, a PA, other broadcasting or audio playback equipment, etc.

[0014] The audio reproduction device 2 is a device that reproduces the transmission signal T generated and transmitted by the audio transmission device. In this embodiment, the audio reproduction device 2 is, for example, a head-mounted display (HMD) for XR (Extended Reality) such as virtual reality (VR), augmented reality (AR), mixed reality (MR), or substitutional reality (SR) that allows head tracking, an ear speaker, earphone, or headphone (hereinafter referred to as "headphone or the like") equipped with a head tracking sensor, an ear-hook type smart device, a stereophonic reproduction device to which headphones or the like are connected, a dedicated game console, a content reproduction device, or other audio reproduction device.

[0015] In this embodiment, the audio transmission device 1 and the audio reproduction device 2 are connected wirelessly or wired. In the case of wireless connection, short-range wireless communication including Bluetooth (registered trademark) and wireless LAN (Wireless Local Area Network, Wi-Fi), Li-Fi or infrared communication, mobile phone networks such as 4G and 5G, medium-range wireless communication, satellite radio communication networks, and other wireless communication technologies may be used. In the case of wired connection, various types of USB (Universal Serial Bus), wired LAN, wired WAN (Wide Area Network), HDMI (registered trademark), DP (DisplayPort), and other dedicated lines may be used for connection. Furthermore, the audio transmission device 1 and the audio reproduction device 2 may be connected via Bluetooth (registered trademark), TCP / IP, UDP, other IP networks, or other communication methods. Furthermore, whether to use wireless or wired connection between the audio transmission device 1 and the audio reproduction device 2 may be selectable depending on the distance between the devices, convenience, etc. Furthermore, the audio transmission device 1 and the audio reproduction device 2 may be connectable in a plurality of ways.

[0016] In this embodiment, binaural audio is generated from an audio signal including channels C-1 to C-n as a source audio signal. Any of the multiple channels C-1 to C-n will be referred to simply as "channel C" below.

[0017] In the audio transmission and reproduction system X, the audio transmission device 1 according to this embodiment divides source audio signals into a first category and a second category. Of these divided signals, only the first category signals are binauralized, and control is performed to generate a binauralized signal (first category audio signal F). Then, a transmission signal T including the binauralized first category audio signal F and the non-binauralized second category audio signal S is transmitted to the audio reproduction device 2.

[0018] On the other hand, the audio reproduction device 2 according to this embodiment acquires the transmission signal T transmitted by the audio transmission device 1 and performs control to binauralize the received second category audio signal S. The audio reproduction device 2 generates a binaural signal by adding up this binauralized second category audio signal S and the first category audio signal F that has already been binauralized when received. The audio reproduction device 2 then outputs this binaural signal.

[0019] That is, in this embodiment, the audio transmission device 1 and the audio reproduction device are controlled to send and receive (transmit) data such as the transmission signal T. A delay of several tens of milliseconds to several hundreds of milliseconds or more may occur in the transmission between the audio transmission device 1 and the audio reproduction device 2. Furthermore, the length of this delay may differ depending on whether the connection is wireless or wired.

[0020] Next, we will explain the functional configuration of each device in the audio transmission and reproduction system X. The audio transmission device 1 according to this embodiment has, as its functional configuration, a binauralization unit 11 (first binauralization unit), a signal generation unit 12, and a transmission unit 13. The audio reproduction device 2 has a signal acquisition unit 20, a binauralization unit 21 (second binauralization unit), a binaural generation unit 22, an audio output unit 23, and a direction transmission unit 24.

[0021] The binauralization unit 11 and the binauralization unit 21 are calculation units that perform binauralization by convolving an HRTF or the like with an audio signal. In this embodiment, the binauralization unit 11 binauralizes only a first category signal among the source audio signals based on listener direction information D received from the audio reproduction device 2, to generate a first category audio signal F. The binauralization unit 21 binauralizes a second category audio signal S, which is a second category signal acquired by the signal acquisition unit 20.

[0022] In this embodiment, the binauralization unit 11 may determine whether the source audio signal is likely to be perceived as auditorily unnatural based on the direction information D of the received listener.

[0023] Here, the binauralization unit 11 may determine, for each channel C, whether or not the azimuth and elevation angle make it easy to perceive auditory unnaturalness. In this case, the binauralization unit 11 may determine, for each channel C, the azimuth and elevation angle based on the listener's direction information D and the direction information set for channel C. Alternatively, the binauralization unit 11 may determine the low-frequency components of channel C of the source audio signal as a first category audio signal F. This low-frequency component may be, for example, an audio component of 1000 to 500 Hz or less divided into frequency bands.

[0024] The signal generating unit 12 generates a transmission signal T including the first category audio signal F binauralized by the binauralizing unit 11 and the second category audio signal S not binauralized.

[0025] The transmission unit 13 transmits in real time the transmission signal T generated by the signal generation unit 12. Specifically, the transmission unit 13 transmits the encoded transmission signal T from the audio transmission device 1 to the audio reproduction device 2 using the various wired or wireless methods described above. In addition, the transmission unit 13 can also receive direction information D of the listener acquired by the audio reproduction device 2.

[0026] The signal acquisition unit 20 acquires the transmission signal T generated by the audio transmission device 1. The signal acquisition unit 20 also acquires the transmission signal T by any of the various wired or wireless methods described above.

[0027] The binaural generation unit 22 generates binaural signals by combining the audio signals of some of the channels with the audio signals of other channels that have been binauralized by the binauralization unit 21 .

[0028] The audio output unit 23 outputs the binaural signal generated by the binaural generation unit 22 as audio for both ears. In this embodiment, the audio output unit 23 includes, for example, a D / A converter and a headphone amplifier, and reproduces audio through headphones of an HMD or the like. Furthermore, the audio output unit 23 may drive vibration elements for bass or a subwoofer. Alternatively, the audio output unit 23 may reproduce the audio signal by encoding it and outputting it as an audio file or streaming audio.

[0029] The direction transmitting unit 24 acquires direction information D of the listener regarding listening to the audio, and transmits it to the audio transmission device 1. Specifically, the direction transmitting unit 24 may acquire the direction information D of the listener from a head tracking device such as an HMD, and transmit it to the audio transmission device 1.

[0030] Here, in this embodiment, the source audio signal can be an audio signal from each direction of the content, an audio signal per content unit, an audio signal of a sound source object, an audio signal divided by frequency band, or multiple audio signals of remote call participants.

[0031] This content may include, for example, various types of content, such as movie and music data, games, audiobooks, e-book data capable of voice synthesis, television and radio broadcast data, various audio data related to operating instructions for car navigation systems and various home appliances, XR content, and other data capable of audio output. Among these, movies may also include musical instrument performances and lectures. Alternatively, content may include background music and sound effects from games, MIDI files, voice call data from mobile phones or transceivers, and synthesized text data from messengers. In this case, Channel C may include, as sound source objects, audio signals derived from objects such as vocals, musical instruments, vehicles, and game characters, as well as human audio signals from actors, narrators, storytellers, storytellers, and other speakers. Channel C may also include audio signals from remote participants in one-to-one, one-to-many, or multiple-to-many live events between locations, video conferencing systems, and the like. The spatial location of these audio signals is set as directional information within the content. Alternatively, the content can include background music and sound effects from games, music files such as MIDI files, voice call data from mobile phones or walkie-talkies, synthesized text data from messengers, and other data that cannot be played directly.

[0032] Alternatively, if the source audio signal is an audio signal from a remote participant, it is also possible to use an audio signal from a microphone at a live venue, an audio signal uttered by a user (participant) of various messengers or video conferencing application software (hereinafter simply referred to as "app") on a PC (Personal Computer) or smartphone, etc. Directional information may include directional information added by a mixer, the orientation of the participant's head in the camera, or the orientation of an avatar placed in a virtual space.

[0033] In either case, an audio signal recorded via a network or a directly connected microphone may also be used as the source audio signal. In this case, directional information may also be added to the audio signal. Alternatively, any combination of the audio signals of the above-mentioned contents and remote participants may be used. Furthermore, directional information may be set for channel C even if it is something other than an audio signal. Furthermore, in this embodiment, this source audio signal may serve as a "target signal" for reproducing the direction for binauralization.

[0034] In this embodiment, direction information may be set in the source audio signal for each channel C. This direction information may be in an absolute coordinate system or a relative coordinate system with respect to the listener. In addition, the type and type of the above-mentioned content may also be set for each channel C.

[0035] In this embodiment, the transmission signal T may be, for example, a signal of various codecs capable of transmitting and receiving audio signals of multiple channels via Bluetooth (registered trademark) in the case of wireless transmission.

[0036] In this embodiment, the first category audio signal F is an audio signal that has been partially binauralized from the source audio signal by the audio transmission device 1. Specifically, the first category audio signal F may be an audio signal of some of the channels C-1 to C-n (hereinafter referred to as "some channels"). More specifically, the first category audio signal F may be a signal that is less likely (less likely) to perceive unnaturalness auditorily due to positional shifts and movements than the second category audio signal S. This signal may be a portion of the source audio signal for which a slow HRTF update speed is not a problem.

[0037] On the other hand, the second category audio signal S is an audio signal that is binauralized from the source audio signal by the audio reproduction device 2, without being binauralized by the audio transmission device 1. Specifically, the second category audio signal S may be an audio signal of a channel different from some of the channels. More specifically, the second category audio signal S may be a signal that is more likely to be perceived as auditorily unnatural with respect to positional deviation and movement than the first category audio signal F. This signal may be a portion of the source audio signal that requires a fast HRTF update speed.

[0038] [Hardware Configuration of Audio Transmission Device 1 and Audio Reproduction Device 2] The audio transmission device 1 and the audio reproduction device 2 include, for example, various circuits such as control means (control units) such as an ASIC (Application Specific Processor), a DSP (Digital Signal Processor), a CPU (Central Processing Unit), an MPU (Micro Processing Unit), and a GPU (Graphics Processing Unit).

[0039] Furthermore, the audio transmission device 1 and the audio reproduction device 2 may include, as storage means (storage unit), a semiconductor memory such as a ROM (Read Only Memory) or a RAM (Random Access Memory), a magnetic recording medium such as a HDD (Hard Disk Drive), an optical recording medium, or the like. The ROM may include a flash memory or other writable or recordable recording medium. Furthermore, an SSD (Solid State Drive) may be provided instead of an HDD. This storage unit may store the control program and various contents according to this embodiment. Of these, the control program is a program for realizing each functional configuration and each method, including the audio signal processing program according to this embodiment. This control program includes an embedded program such as firmware, an OS (Operating System), and an application.

[0040] The content in the storage unit according to this embodiment may be acquired by downloading a file or a data block transmitted by wire or wirelessly, or may be acquired stepwise by streaming, etc. Furthermore, the app according to this embodiment may be an app such as a media player that plays content, or an app for messenger or video conferencing, etc.

[0041] In addition, the audio transmission device 1 and the audio reproduction device 2 may be equipped with a direction calculation means including a GNSS (Global Navigation Satellite System) receiver that calculates the direction in which the listener is facing, an in-room position and direction detector, an acceleration sensor, a gyro sensor, a geomagnetic sensor, etc. that are capable of head tracking, and a circuit that converts the output of these sensors into direction information.

[0042] Furthermore, the audio transmission device 1 and the audio reproduction device 2 may include a display unit such as a liquid crystal display, an organic EL display, or an LED (Light Emitting Diode), an input unit such as a button, a keyboard, a pointing device such as a mouse or a touch panel, and an interface unit for connecting to various devices wirelessly or via a wired connection. Of these, the interface unit may include an interface such as a flash memory medium such as a micro SD (registered trademark) card or a USB (Universal Serial Bus) memory, a LAN board, a wireless LAN board, a serial interface, a parallel interface, etc.

[0043] Furthermore, the audio transmission device 1 and the audio reproduction device 2 can realize the methods according to the present embodiment using hardware resources by having the control means execute various programs stored mainly in the storage means. These various programs include the audio signal processing program according to the present embodiment. Note that a part or any combination of the above-described configurations may be configured in terms of hardware or circuits using ICs, programmable logic, FPGAs (Field-Programmable Gate Arrays), etc.

[0044] [Audio Transmission and Reproduction Processing by the Audio Reproduction Device 2] Next, with reference to FIGS. 2 and 3, an audio transmission and reproduction processing by the audio reproduction device 2 according to the embodiment of the present invention will be described.

[0045] First, an overview of the audio transmission and reproduction process according to this embodiment will be described with reference to Fig. 2. In the audio transmission and reproduction process according to this embodiment, audio signals (source audio signals) of multiple channels are divided into first category audio signals F, for which a slow HRTF update rate is acceptable, and second category audio signals S, for which a fast HRTF update rate is required. The first category audio signals F are then binauralized by the audio transmission device 1. Meanwhile, the second category audio signals S are binauralized by the audio reproduction device 2. By updating the second category signals S, for which a fast update rate is required, after receiving communication, there is no delay in updating the head-related transfer functions corresponding to head movement for the second category signals S, even if there is a delay in communication time, and it is possible to prevent unnatural sound images.

[0046] In the audio transmission and playback process of this embodiment, the control means of each of the audio transmission device 1 and the audio playback device 2 cooperates with each unit to control and execute a control program stored in a storage means using hardware resources, or the control program is processed directly by each circuit. Below, the details of the audio transmission and playback process will be explained step by step with reference to the flowchart in Figure 3.

[0047] (Step S101) First, the binauralization unit 11 of the audio transmission device 1 performs audio separation processing. The binauralization unit 11 determines, for each channel C of the source audio signal, whether or not auditory unnaturalness is likely to be perceived when the update of the head-related transfer function in response to head movement is delayed. Then, the binauralization unit 11 separates the channel C for which auditory unnaturalness is not likely (is hard to perceive) as a first category audio signal F. In this embodiment, the binauralization unit 11 separates the source audio signal into a portion for which the HRTF update speed must be fast and a portion for which a slower update speed is acceptable.

[0048] Specifically, the binauralization unit 11 receives listener direction information D from the audio reproduction device 2 via the transmission unit 13. Furthermore, the binauralization unit 11 can acquire the direction of the audio in the virtual space for each channel C. Then, based on the listener direction information D, the binauralization unit 11 calculates the directions (azimuth and elevation) of the channel C and the listener in the spatial arrangement including the virtual space.

[0049] Here, the binauralization unit 11 determines whether or not the azimuth and elevation angle are likely to cause auditory perception of unnaturalness. As a specific example, the binauralization unit 11 may separate, as a first category audio signal F, a channel C signal whose absolute values ​​of the azimuth and elevation angle relative to the listener are equal to or greater than a specific angle threshold. This angle threshold can be set separately for each of the azimuth and elevation angle based on experimental values, etc. On the other hand, the binauralization unit 11 may separate, as a second category audio signal S, a channel C signal whose absolute values ​​of the azimuth and elevation angle are less than a specific angle threshold.

[0050] In this example, the binauralization unit 11 determines that for a channel C signal whose direction relative to the listener is upward, downward, or to the side, positional shifts and movements are not likely (hardly) perceived as auditory unnatural. Therefore, the binauralization unit 11 separates this channel C signal from the first category audio signal F and binauralizes it before transmission by the audio transmission device 1. Conversely, the binauralization unit 11 determines that when the direction relative to the listener is close to the front, positional shifts and movements are likely to be perceived as auditory unnatural. Therefore, the binauralization unit 11 separates this channel C signal as the second category audio signal S.

[0051] For example, the binauralization unit 11 may separate a channel C of a musical instrument performance as a first category audio signal F. This is because instrument sounds have a broad range of components coming from the sides, and therefore, it is expected that tracking by head tracking can be performed relatively slowly. On the other hand, the binauralization unit 11 may separate a channel C of vocals as a second category audio signal S. This is because vocals are often located horizontally in front of the listener, and therefore auditory perception is sensitive to their positional shifts and movements. In other words, binauralization by the audio reproduction device 2 enables high-speed tracking of the listener's head movement. Here, the binauralization unit 11 may determine the direction of channel C in the virtual space based on the content and type of channel C's content, or may determine the direction based on channel C's setting information.

[0052] Alternatively, the binauralization unit 11 may determine and separate, for each channel C of the source audio signal, the low-frequency components as the first category audio signal F. In this case, the binauralization unit 11 may determine and separate, the high-frequency components as the second category audio signal S. In other words, the binauralization unit 11 may separate, the low-frequency components as the first category audio signal F and the high-frequency components as the second category audio signal S.

[0053] In this case, the binauralization unit 11 may separate each channel C into a channel for low-frequency components and a channel for high-frequency components using a band-pass filter such as a low-pass filter or a high-pass filter. Then, the binauralization unit 11 may determine the channel for the low-frequency components as the first category audio signal F. Furthermore, the binauralization unit 11 may determine the channel for the high-frequency components as the second category audio signal S. Alternatively, the binauralization unit 11 may calculate the ratio between the low-frequency components and the high-frequency components for each channel C and determine whether the channel is a first category audio signal F or a second category audio signal S based on this ratio.

[0054] In this example, the binauralizer 11 may separate the low-frequency components as a first category audio signal F to be binauralized by the audio transmission device 1 because positional shifts and movements are less likely to be perceived as unnatural auditorily. On the other hand, the binauralizer 11 may separate the high-frequency components as a second category audio signal S to be binauralized by the audio reproduction device 2 because positional shifts and movements are more likely to be perceived as unnatural auditorily. In this case, since the high-frequency components have little energy, a low bit rate can be achieved even when encoded in multiple channels. On the other hand, since the high-frequency components play an important role in directional perception, binauralization by the audio reproduction device 2 makes it possible to quickly follow the head movement of the listener.

[0055] (Step S102) Next, the binauralization unit 11 performs a first binauralization process. Here, the binauralization unit 11 performs binauralization (two-channel left and right) by convolving HRTFs and the like for each channel of the first category audio signal F based on sound source direction information and listener direction information D. That is, the binauralization unit 11 binauralizes the first category audio signal F that is determined to be not easily (hard to be) perceived as auditorily unnatural even with delays due to transmission, etc. In the above example, by separating many instrument sounds as the first category audio signal F and binauralizing them (two-channel), it is possible to save transmission bandwidth during encoding.

[0056] On the other hand, the binauralization unit 11 does not binauralize the second category audio signal S that is determined to be auditorily unnatural. That is, since the second category audio signal S is an audio signal that is auditorily unnatural due to delay or the like, it is binauralized by the binauralization unit 21 provided in the audio reproduction device 2 that is closer to the listener. That is, as will be described later, this second category audio signal S is binauralized by the audio reproduction device 2 after transmission, which will be described below.

[0057] (Step S103) Next, the signal generation unit 12 performs a transmission signal generation process. The signal generation unit 12 generates a transmission signal T including a first category audio signal F that has already been binauralized and a second category audio signal S that has not been binauralized. For example, when transmitting a signal via Bluetooth (registered trademark), the signal generation unit 12 may encode the first category audio signal F and the second category audio signal S using various codecs such as AAC, AptX LL, HD, LDAC, and LHDC, and include the encoded signal in the transmission signal T. Furthermore, the signal generation unit 12 may encode audio signals of multiple channels, such as 2 channels (2ch), 5.1 channels (5.1ch), 7.1 channels (7.1ch), and 22.2 channels (22.2ch), in accordance with audio encoding methods such as MPEG-2 AAC, MPEG-4 AAC, MP3, Dolby (registered trademark), and DTS (registered trademark), as the transmission signal T. In addition, the signal generation unit 12 may bundle these audio signals of multiple channels, control signals, and the like, into packets or the like to generate the transmission signal T.

[0058] (Step S104) Next, the transmission unit 13 performs a transmission signal transmission process. The transmission unit 13 transmits the transmission signal T in real time. At this time, the transmission unit 13 transmits the transmission signal T by various methods such as wired or wireless. This makes it possible to perform a "hybrid" transmission of the binauralized first category audio signal F and the non-binauralized second category audio signal S.

[0059] (Step S201) Here, we will explain the processing of the audio reproduction device 2. The signal acquisition unit 20 of the audio reproduction device 2 performs signal acquisition processing. The signal acquisition unit 20 receives (acquires) the transmission signal T generated by the audio transmission device 1 and temporarily stores it in a storage means.

[0060] (Step S202) Next, the binauralization unit 21 performs a second binauralization process. The binauralization unit 21 binauralizes the second category audio signal S. As described above, the second category audio signal S is an audio signal that is likely to be perceived as auditorily unnatural.

[0061] (Step S203) Next, the binaural generation unit 22 performs binaural signal generation processing. The binaural generation unit 22 generates binaural signals by combining audio signals of some channels with audio signals of other channels that have been binauralized by the binauralization unit 21. In this embodiment, the binaural signals may include an L signal for the left ear and an R signal for the right ear of the listener. Furthermore, the binaural signals may be digital signals that can be heard by the listener by decoding digital data and playing them back at the audio output unit 23.

[0062] (Step S204) Next, the audio output unit 23 performs audio output processing. The audio output unit 23 outputs the audio signals generated by the binaural generation unit 22. This output may be, for example, a 2-channel analog audio signal corresponding to the left and right ears of the listener. This allows the audio output unit 23 to reproduce audio signals corresponding to a virtual sound field as 2-channel audio signals through headphones. This allows the listener to hear left and right sounds in a stereophonic sound.

[0063] (Step S205) Next, the direction transmission unit 24 performs a direction transmission process. The direction transmission unit 24 can acquire the head direction of the listener from head tracking using a gyro sensor or the like in an HMD or smartphone, and direction information such as the orientation of an avatar in a virtual space. The direction transmission unit 24 transmits the acquired direction information D of the listener to the audio transmission device 1. At this time, the direction transmission unit 24 may directly transmit the direction information D of the listener via a wired or wireless connection, separately from the transmission signal T. Alternatively, the direction transmission unit 24 may be configured to transmit the direction information D of the listener via the signal acquisition unit 20 via a wireless or wired connection such as Bluetooth (registered trademark), the same as the transmission signal T. This completes the audio transmission and playback process according to this embodiment.

[0064] The above-described configuration can achieve the following effects. In recent years, it has become common to connect an HMD or headphones to a smartphone to enjoy a virtual sound field (3D sound) for VR or AR. However, in conventional headphone playback of 3D sound, in order to generate binaural signals, convolution processing of HRIRs of the corresponding sound source directions was performed on the transmitting side, such as a smartphone, with multiple sound source signals. In other words, out-of-head localization was achieved and 3D sound was generated by convolving the head-related transfer function of the sound source direction with the audio signal of each channel of each source audio signal to generate binaural signals.

[0065] In this case, in a head-tracking system, the relative direction between the sound source and the listener changes when the listener moves their head, so the system detects the listener's head direction, transmits this direction information to a smartphone or the like, and updates the HRIR accordingly. That is, a stationary sound source is generated by calculating the direction of the HRTF or the like to be used from the relationship between the listener's head direction and the sound source position and instantly updating it. The same processing is performed for moving sound sources, and a moving sound source is generated in accordance with the listener's head movement.

[0066] However, delays in communication between smartphones and headphones have been a problem. Specifically, with wireless headphones using Bluetooth (registered trademark) or the like, which have significant delays, it takes a total of about 200 milliseconds to transmit directional information detected by head tracking to a smartphone and then to transmit information such as HRTF updated based on that information to the headphones. In other words, the update of the HRTF is delayed by 200 milliseconds or more relative to head movement.

[0067] For this reason, conventional headphones with head tracking have a problem in that they are poor at tracking the listener's head movement, resulting in an unstable sound image relative to the listener's movement. In other words, a delay equivalent to the round-trip communication time occurs in updating HRTFs, etc., making the sound image unnatural due to the listener's head movement. Specifically, when the listener rotates their head, a phenomenon occurs in which the sound image of a stationary sound source temporarily follows the head rotation. In fact, this problem has been observed in consumer earphones and other devices with head tracking. Unless this delay is approximately 60 milliseconds or less, the unnaturalness of the sound image localization is significantly perceived.

[0068] In contrast, (A) a signal transmission method according to an embodiment of the present invention is a method for transmitting a transmission signal T including audio signals of multiple channels, characterized in that the transmission signal T including a binauralized first category audio signal F and a non-binauralized second category audio signal S is transmitted in real time.

[0069] This provides a signal transmission method for transmitting audio signals containing three or more channels C, in which some of the signals are binauralized as a first category and the rest are transmitted as a second category without binauralization. By dividing the binauralization process between the transmitting and receiving sides, the second category audio signals can be binauralized at the listener's side, reducing the unnaturalness of the sound image. In other words, the effects of transmission delays can be minimized, making it less likely that the listener will perceive an unnatural sound. This makes the method applicable to various XR application software, such as games, movies, and home TV.

[0070] In addition, conventionally, binauralization of multi-channel audio signals for headphone playback has been performed by a transmitting device such as a smartphone. In this case, the audio signals to be transmitted are two-channel binaural signals, so degradation of sound quality due to encoding for transmission has been minimized. However, as described above, this has caused a problem in which the sound image becomes unnatural due to a delay in sound image tracking caused by a transmission delay.

[0071] On the other hand, to eliminate the unnaturalness of the sound image due to such delays, if binaural processing is performed on the receiving side, such as headphones, rather than on the transmitting side, delays in updating the HRTF due to communication do not occur, and the HRTF can be updated instantly according to the listener's head direction. However, in such a case, it is necessary to transmit the multi-channel audio signals to the receiving side as multi-channel signals before converting them into binaural signals. This increases the transmission rate (transmission capacity), making it necessary to perform highly efficient encoding with a high compression rate. This could lead to degradation of sound quality due to encoding distortion.

[0072] In contrast, the signal transmission method according to an embodiment of the present invention can instantaneously track head movement while transmitting without degradation of encoding. In other words, the signal-to-noise ratio (hereinafter referred to as "SNR") of the audio can be increased. Therefore, by applying the signal transmission method according to this embodiment to a 3D audio playback system, such as a wireless headphone playback system connected to a smartphone, playback of higher quality audio than before becomes possible. Furthermore, by applying the signal transmission method to smartphones, home appliances, etc., it is possible to generate high-quality stereophonic audio from multi-channel audio signals, making it applicable to international standardization, etc.

[0073] (B) In a signal transmission method according to an embodiment of the present invention, the binauralized first category audio signal F is an audio signal of some channels, and the audio signal that is not binauralized is an audio signal of other channels, which is a signal transmission method described in (A).

[0074] In this configuration, in signal transmission of an audio signal including three or more channels C, the first category channels C are binaurally processed before transmission, and the second category channels C are binaurally processed after reception, and these are summed on the receiving side to generate a binaural signal. That is, the source audio signal may be divided into two categories, first category audio signals F and second category audio signals S, on a channel C basis, and dynamic bit allocation may be performed between these two categories. That is, the allocation of transmission bandwidth may be dynamically changed. In this way, by performing hybrid transmission in which the binaural processing is shared between the transmitting side and the receiving side, audio signals can be transmitted with high sound quality while suppressing unnatural sound field artifacts caused by transmission delays.

[0075] In this case, for example, a configuration can be realized in which a channel C belonging to a first category audio signal F is binauralized by an audio transmission device 1 before encoding, and a channel C belonging to a second category audio signal S is encoded on a channel-by-channel basis and then binauralized after being decoded by an audio reproduction device 2.

[0076] (C) In a signal transmission method according to an embodiment of the present invention, the second category audio signal S is a signal that is more likely to be perceived as auditorily unnatural due to positional shifts and movements than the first category audio signal F, and is characterized by being a signal transmission method described in (A) or (B).

[0077] With this configuration, audio signals of multiple channels are divided into first category audio signals F, for which a slow HRTF update rate is acceptable because unnaturalness due to a positional shift in the sound source direction is not easily (hardly) perceived auditorily, and second category audio signals S, for which a fast HRTF update rate is required because unnaturalness due to a positional shift in the sound source direction is easily perceived auditorily. Then, only the first category audio signals F, for which unnaturalness due to a positional shift in the sound source direction is not easily (hardly) perceived auditorily, are binauralized by the audio transmission device 1, and the second category audio signals S, for which unnaturalness is easily perceived auditorily, are binauralized by the audio reproduction device 2. This reduces the number of channels of the transmitted signal to minimize coding distortion, while realizing high-speed HRTF update for necessary sound sources, thereby achieving high-quality sound and stable sound field reproduction with stable positioning even when head movement is present.

[0078] (D) A signal transmission method according to an embodiment of the present invention is characterized in that the source audio signal, which is an audio signal of multiple channels, is a signal transmission method described in (C), in which whether or not it is likely to be perceived as auditory unnatural is determined based on the azimuth and elevation angle.

[0079] With this configuration, it is possible to group each channel C into two categories according to the azimuth angle and elevation angle. Therefore, for example, a musical instrument sound having a wide range of components coming from the sides can be determined as a first category audio signal F that is less likely to be perceived as unnatural to the auditory sense, since it can be relatively slow to follow. Therefore, by converting musical instrument sounds into binaural sounds using the audio transmission device 1, it is possible to reduce the signal capacity for transmission. In other words, by converting many musical instrument sounds into binaural sounds and leaving them only in left and right channels (2ch), it is possible to significantly reduce the capacity required for encoding.

[0080] On the other hand, for example, vocals, which are often located horizontally in front of the listener, are easily perceived as unnatural to the ear, and can be determined to be a second-category audio signal S that is sensitive to positional shifts. For this reason, only the second-category audio signal S of the vocals is transmitted as an independent signal, and the first-category audio signal F of the musical instrument performance is binauralized in advance and transmitted. Then, by binauralizing the second-category audio signal S using the audio reproduction device 2, high-speed tracking of head movement can be achieved.

[0081] (E) A signal transmission method according to an embodiment of the present invention is characterized in that the first category audio signal F is a low-frequency component and the second category audio signal S is a high-frequency component, which is a signal transmission method described in (C).

[0082] This configuration allows for grouping each channel C into two categories according to the transmission bandwidth. Therefore, for example, it is possible to binaurally process low-frequency components before transmission and then transmit them, and binaurally process high-frequency components after reception, and then combine them at the receiving end to generate binaural signals. Specifically, it is possible to binaurally process low-frequency components of musical instrument sounds before transmission, and encode only the high-frequency components in an independent channel and then binaurally process them at the receiving end. In this case, since the high-frequency components of the audio signal have little energy, a low bit rate (transmission bandwidth) can be achieved even when encoding multiple channels. On the other hand, high-frequency components are important for the listener's directional perception, and therefore are preferable because they need to track head movement at high speed.

[0083] (F) A signal generation method according to an embodiment of the present invention is an audio generation method for generating a transmission signal T including audio signals of multiple channels, characterized in that only a first category audio signal F is binauralized based on received listener direction information D, and a transmission signal T is generated that includes the first category audio signal F binauralized by the binauralization unit 11 and a second category audio signal S that has not been binauralized.

[0084] With this configuration, in signal transmission of an audio signal including three or more channels C, an audio transmission device 1 such as a smartphone can binauralize only the first category audio signal F using directional information D of the listener, which may be delayed.

[0085] (G) A signal reproduction method according to an embodiment of the present invention is a method for reproducing a transmission signal T including audio signals of multiple channels, and is characterized in that it acquires a transmission signal T transmitted by a signal transmission method described in any one of (A) to (E) or a transmission signal T generated by a signal generation method described in (F), binauralizes the received second category audio signal S, and generates a binaural signal by combining the binauralized second category audio signal S with a first category audio signal F that has already been binauralized when received.

[0086] With this configuration, in signal reproduction of an audio signal containing three or more channels, some channels C of the signal group can be binauralized before transmission, and the other channels C can be binauralized after reception, and then these can be summed to generate a binaural signal. That is, in signal reproduction of a source audio signal, some of the received transmission signals T can be binauralized as received, and some of the received transmission signals can be binauralized after reception, and then these can be summed to generate a binaural signal. This minimizes coding distortion of the transmission signals T for source audio signals such as content, point-to-point connections, point-to-multipoint connections, multipoint-to-multipoint connections, and remote conferences, while realizing sound field reproduction that is less likely to cause a listener to perceive unnaturalness. Furthermore, the binaural audio signals can be output through headphones, an HMD, or the like, to experience realistic sound.

[0087] (H) An audio transmission and reproduction system X according to an embodiment of the present invention is a reproduction system including an audio transmission device 1 that generates a transmission signal T including audio signals of multiple channels, and an audio reproduction device 2 that reproduces the transmission signal T generated by the audio transmission device. The audio transmission device 1 includes a binauralizer 11 that binauralizes only a first category audio signal F based on direction information D of the listener received from the audio reproduction device 2, a signal generator 12 that generates a transmission signal T including the first category audio signal F binauralized by the binauralizer 11 and a second category audio signal S that is not binauralized, and The audio reproduction device 2 is characterized by comprising a signal acquisition unit 20 that acquires the transmission signal T generated by the audio transmission device 1, a binauralization unit 21 that binauralizes the second category audio signal S acquired by the signal acquisition unit 20, a binaural generation unit 22 that generates a binaural signal by combining the second category audio signal S binauralized by the binauralization unit 21 and the second category audio signal F, an audio output unit 23 that outputs the audio signal generated by the binaural generation unit 22, and a direction transmission unit 24 that transmits direction information D of the listener to the audio transmission device 1.

[0088] By configuring in this way, it is possible to transmit the signal with the coding distortion of the signal being transmitted minimized, while minimizing the effects of transmission delay, making it difficult for listeners to perceive unnaturalness.

[0089] [Other Embodiments] In the above-described embodiment, an example has been described in which each of the channels C-1 to C-n of a source audio signal is directly divided into a first-category audio signal F and a second-category audio signal S on a channel C basis, and an example in which the signals are divided based on either low-frequency components or high-frequency components. However, the first-category audio signal may be obtained by clustering or panning the channels C-1 to C-n of the source audio signal, which are several to several hundred (n) signals present in a 3D (three-dimensional) space, in m specific directions, which is less than the number n of channels. In this case, the signals for the m directions may be generated within the audio transmission device 1 by applying an appropriate gain to each signal of the channels C-1 to C-n and calculating the sum of the signals time-shifted, and then combined as a representative point signal present at a representative point in the m directions. This combined representative point signal may then be transmitted to the audio reproduction device 2. In other words, in this example, the signals for the m directions (representative point signals) may be treated as the first-category audio signal. Furthermore, the signals of the m directions (representative point signals) may be binauralized by convolving HRIR or the like in the audio reproduction device 2. Additionally, in this case, the second category audio signal that is not binauralized may be transmitted together with the first category audio signal as a signal for transmission. Alternatively, the second category audio signal in this case may be a control signal for encoding the audio signal, or the like.

[0090] In this case, as in the above-described embodiment, a delay may occur in transmission between the audio transmission device 1 and the audio reproduction device 2. The audio reproduction device 2 that receives the transmission signal may combine (binauralize) the m signals while updating the HRIR in real time according to the head direction information and the orientations of the m signals in accordance with the movement of the listener's head. In this case, the audio transmission device 1 may sum the signals distributed to the representative point of channel C to be combined at the representative point, generate a sum signal, and transmit this sum signal. Then, the binauralization unit 21 of the audio reproduction device 2 may convolute the HRIR at the position of the representative point (HRIR in the direction of the representative point) with this sum signal to binauralize it and generate binaural signals.

[0091] That is, in a signal transmission method according to another embodiment of the present invention, the first category audio signal may be a representative point signal obtained by grouping a signal (source audio signal) including audio signals of multiple channels in a direction that has fewer channels than the source audio signal. Alternatively, in a signal reproduction method according to another embodiment of the present invention, the audio transmission device 1 may generate a representative point signal obtained by grouping a source audio signal in a direction that has fewer channels than the source audio signal, transmit the representative point signal in real time, and the audio reproduction device 2 may binaurally convert the representative point signal to generate a binaural signal.

[0092] With this configuration, the audio transmission device 1 aggregates several to several hundred (n) signals that require HRIR convolution in XR, etc., into m directional signals, which are then transmitted to the audio reproduction device 2 and binaurally processed. Aggregating m signals reduces the transmission bandwidth, enabling easy wireless transmission. Furthermore, compared to conventional methods in which HRIRs are individually convolved with each sound source signal, the amount of computation can be reduced. Additionally, binaural processing by the audio reproduction device 2 enables the orientation of the virtual speaker, which is the representative direction of panning, to be controlled in conjunction with the listener's head movement. This allows the sound source position to be updated instantaneously in response to head movement without being affected by transmission delays. Therefore, the HRIR can be updated instantaneously in response to head movement, eliminating delays. In other words, reducing the number of sound source objects to be reproduced by panning enables wireless transmission from a smartphone, etc. to headphones, etc. This allows for extremely realistic HMDs, wireless headphones, etc.

[0093] In the above-described embodiment, the angular directions of the listener in the up, down, left, and right directions are considered as directional information. However, the directional information may be configured to consider only the left and right directions, or may be configured to consider six degrees of freedom (6 DOF).

[0094] In the above-described embodiment, the judgment of whether or not the sound is likely to be perceived as auditory unnatural is performed using either the judgment based on the azimuth and elevation angle or the judgment based on the low-frequency components and the high-frequency components. However, the binauralization unit 11 may combine the judgment based on the azimuth and elevation angle with the judgment based on the low-frequency components and the high-frequency components. For example, the vocal channel C may be generated as a second-category audio signal S, and the low-frequency components of the instrumental sound channel C may be generated as a first-category audio signal F, and the high-frequency components may be separated as a second-category audio signal S into a channel separate from the vocal channel C.

[0095] In the above-described embodiment, bit allocation, i.e., allocation of transmission bandwidth, is performed according to the energy of each component when separating the first category audio signal and the second category audio signal. However, the bandwidth of the transmission signal may be different between the first category audio signal and the second category audio signal. For example, the bandwidth of the first category audio signal may be narrower than that of the second category audio signal. In this case, the bandwidth of the high-frequency components may be set according to the amount of energy, as in the above-described embodiment, while only the bandwidth of the low-frequency components may be reduced. Alternatively, the ratio of the bandwidth between the first category audio signal and the second category audio signal may be changed based on the moving speed or acceleration of the listener's directional information D. Alternatively, the ratio of the bandwidth between the first category audio signal and the second category audio signal may be changed based on whether the connection is wireless or wired, or simply the magnitude of the transmission delay. In this case, if the delay is less than several tens of milliseconds due to a high-speed wired connection, separation into the first category audio signal and the second category audio signal may not be necessary.

[0096] By configuring in this way, it is possible to save on the transmission band, make it difficult for the listener to notice the positional deviation of the sound source direction, and enable high-quality audio transmission.

[0097] In the above-described embodiment, the binaural conversion is described as convolving the HRIR with the audio signal of channel C. However, similar processing can be performed by converting the audio signal of channel C into the frequency domain and applying the HRTF. Furthermore, it is also possible to configure the system so that the low-frequency components and the high-frequency components are separated and the HRTF is applied only in the case of hybrid transmission. Specifically, as in the above-described embodiment, by using HRTFs for the low and high frequency ranges based on frequencies near or slightly higher than the frequency band where human auditory sensitivity is high, more accurate synthesis is possible.

[0098] Furthermore, although the above-described embodiment does not describe differences in HRTFs, etc., in practice, the binauralization unit 21 may be able to select, from an HRIR table, a user's personal HRIR or an HRIR generated from an HRIR database. Furthermore, when the speaker and listener are transformed into avatars in a virtual space, the binauralization unit 21 can also select HRIRs accordingly. That is, for example, in the case of an avatar shaped like a cat or rabbit with ears pointing upward, it is possible to select HRIRs that sound similar to this.

[0099] Furthermore, the binauralizer 21 can further enhance the sense of reality by separately superimposing the direct sound of channel C and the reflected sound from the environment using convolution, etc. This configuration makes it possible to reproduce clearer sounds that are closer to reality.

[0100] Additionally, in the above embodiment, an example has been described in which the audio output unit 23 reproduces two left and right channels. However, it is also possible to configure the audio to be reproduced using headphones or the like that can reproduce multiple channels as separate channels. Furthermore, the binaural sound generation unit 22 may be capable of separately generating and reproducing signals for a subwoofer or vibration.

[0101] In the above-described embodiment, the audio reproduction device 2 is described as being configured as an integrated device. However, the audio reproduction device 2 may be configured so that a device for receiving audio is connected to a device for actually reproducing audio, such as a headset, headphones, or left-right separated earphones. That is, the audio reproduction device 2 itself may be configured from multiple devices separate from the audio transmission device 1. In addition, the audio transmission device 1 itself may be configured from multiple devices. Furthermore, although the above-described embodiment describes an example in which the distance between the audio transmission device 1 and the audio reproduction device 2 is relatively short, this distance may also be long. In this case, the functions of the audio transmission device 1 may be executed by a server on an intranet or the Internet.

[0102] It is also possible to configure the audio playback device 2 to play content stored on an optical medium or a flash memory card, generated sound source objects, etc. as source audio signals, and at this time, to transmit the first category audio signal F from this source audio signal to the audio transmission device 1 to convert it into binaural audio.

[0103] In the above-described embodiment, the audio reproduction device 2 is described as including the binaural generation unit 22 and the audio output unit 23. However, a configuration that does not include the binaural generation unit 22 or the audio output unit 23 is also possible. In this case, for example, data of the generated audio signal, such as a binaural signal, may be stored on a recording medium. In this case, the other audio reproduction device is applicable to all devices that can acquire the direction of channel C in a virtual space, such as devices equipped with a television or display, videophone calls through a display, videoconferencing, telepresence, etc.

[0104] In the above embodiment, an example has been described in which directional information is added to the audio signal of channel C. However, in a situation where a conversation is taking place in which the speaker and listener are constantly changing roles, such as in the above-mentioned remote conference, directional information does not need to be added to the audio signal of channel C. In other words, when the current listener is the speaker, the direction of the speaker (current listener) can be estimated using the spoken audio signal, and this can be used as the direction of the listener as seen from the current speaker.

[0105] In this case, the direction transmitting unit 24 calculates the arrival direction of the audio signal of the left (L) channel signal (hereinafter referred to as the "L signal") and the right (R) channel signal (hereinafter referred to as the "R signal") as seen from the listener. In this case, the direction transmitting unit 24 may obtain the intensity ratio between the L channel and the R channel. It is also possible to estimate the arrival direction of the signal of each frequency component from the intensity ratio.

[0106] Alternatively, the direction transmitting unit 24 may estimate the direction of arrival of the audio signal from the relationship between the ITD (Interaural Time Difference) of signals of each frequency in HRTF etc. and the direction of arrival. The direction transmitting unit 24 may refer to the relationship between the ITD and the direction of arrival stored in the storage unit as a database.

[0107] Alternatively, it is also possible to estimate the direction of a speaker or listener by performing face recognition from facial image data of a person, such as a speaker or listener in content or a video conference. That is, direction estimation is possible even in a configuration without head tracking. Similarly, it may be possible to grasp the position of a speaker or listener in space. This configuration makes it possible to accommodate various flexible configurations. Furthermore, in an XR in which the sound source position is set in advance, it may be possible to obtain the direction of channel C from the positional relationship between channel C and listener L without estimating the sound source direction.

[0108] It goes without saying that the configurations and operations of the above-described embodiments are merely examples, and can be modified as appropriate within the scope of the present embodiments.

[0109] The audio transmission device of this embodiment can provide an audio transmission and reproduction system that prevents sound images from becoming unnatural when generating stereophonic sound, and can be used industrially.

[0110] REFERENCE SIGNS LIST 1 Audio transmission device 2 Audio reproduction device 11, 21 Binaural conversion unit 12 Signal generation unit 13 Transmission unit 20 Signal acquisition unit 22 Binaural generation unit 23 Audio output unit 24 Direction transmission unit D Direction information of listener F First category audio signal S Second category audio signal T Transmission signal C-1 to C-n Channels X Audio transmission reproduction system

Claims

1. A method for transmitting a signal including a multi-channel audio signal, the method comprising transmitting in real time the signal including a first category audio signal that has been binauralized and a second category audio signal that has not been binauralized.

2. The signal transmission method according to claim 1, characterized in that the first category audio signals that have been binauralized are audio signals of some channels, and the second category audio signals that have not been binauralized are audio signals of other channels.

3. A signal transmission method according to claim 1, characterized in that the second category audio signal is a signal which is more likely to be perceived as auditorily unnatural with respect to positional deviation and movement than the first category audio signal.

4. The signal transmission method according to claim 3, characterized in that the audio signals of the multiple channels are judged as to whether they are likely to be perceived as auditorily unnatural depending on the azimuth and elevation angle.

5. A signal transmission method according to claim 3, characterized in that the first category voice signal is a low-frequency component, and the second category voice signal is a high-frequency component.

6. The signal transmission method according to claim 1, characterized in that the first category audio signal is a representative point signal in which a signal including audio signals of the multiple channels is grouped in a direction smaller than the number of channels of the signal.

7. A method for generating a signal including audio signals of multiple channels, comprising the steps of: binauralizing only a first category audio signal based on received directional information of a listener; and generating the signal including the binauralized first category audio signal and a second category audio signal that is not binauralized.

8. A method for reproducing a signal including audio signals of multiple channels, comprising the steps of: acquiring a signal transmitted by a signal transmission method as defined in any one of claims 1 to 6, or a signal generated by a signal generation method as defined in claim 7; binauralizing the received second category audio signal; and generating a binaural signal by combining the binauralized second category audio signal with the first category audio signal that has already been binauralized when received.

9. A signal reproduction method executed by an audio generation and reproduction system including an audio transmission device that generates and transmits a signal including audio signals of multiple channels, and an audio reproduction device that reproduces the signal transmitted from the audio transmission device, characterized in that the audio transmission device generates a representative point signal that groups the signals including the audio signals of multiple channels in a direction that is fewer than the number of channels of the signals, and transmits the representative point signal in real time; and the audio reproduction device binauralizes the representative point signal to generate a binaural signal.

10. An audio signal processing program executed by an audio transmission device, the audio transmission device binauralizing only a first category audio signal based on received directional information of a listener, and generating a signal including the binauralized first category audio signal and a second category audio signal that has not been binauralized.

11. An audio signal processing program executed by an audio playback device, the audio signal processing program causing the audio playback device to acquire a signal transmitted by a signal transmission method described in any one of claims 1 to 6 or a signal generated by a signal generation method described in claim 7, binauralize the received second category audio signal, and generate a binaural signal by combining the binauralized second category audio signal with the first category audio signal that has already been binauralized when received.

12. An audio transmission device that generates a signal including audio signals of multiple channels, comprising: a first binauralization unit that binauralizes only a first category audio signal based on directional information of a listener received from an audio playback device; and a signal generation unit that generates the signal including the first category audio signal binauralized by the first binauralization unit and a second category audio signal that is not binauralized.

13. A playback device that plays back a signal including audio signals of multiple channels, comprising: a signal acquisition unit that acquires the signal including a binauralized first category audio signal and a second category audio signal that is not binauralized; a second binauralization unit that binauralizes the second category audio signal acquired by the signal acquisition unit; a binaural generation unit that generates a binaural signal by adding the second category audio signal binauralized by the second binauralization unit and the first category audio signal; and an audio output unit that outputs the binaural signal generated by the binaural generation unit.

14. A sound generation and playback system including an audio transmission device that generates and transmits a signal including audio signals of multiple channels, and an audio playback device that plays back the signal transmitted from the audio transmission device, wherein the audio transmission device comprises: a first binauralization unit that binauralizes only a first category audio signal based on directional information of a listener received from the audio playback device; a signal generation unit that generates the signal including the first category audio signal binauralized by the first binauralization unit and a second category audio signal that is not binauralized; and a transmission unit that transmits the signal generated by the signal generation unit in real time, and the audio playback device comprises: a signal acquisition unit that acquires the signal generated by the audio transmission device; a second binauralization unit that binauralizes the second category audio signal acquired by the signal acquisition unit; a binaural generation unit that generates a binaural signal by adding up the second category audio signal binauralized by the second binauralization unit and the first category audio signal; and an audio output unit that outputs the binaural signal generated by the binaural generation unit. a direction transmitting unit that transmits direction information of the listener to the audio transmitting device.

Images