Signal transmission method, signal generation method, signal reproduction method, sound signal processing program, sound transmission device, sound reproduction device, and sound transmission and reproduction system

Abstract

To provide a signal transmission method which suppresses unnaturalness of a sound image, a signal generation method, a signal reproduction method, a sound signal processing program, a sound transmission device, a sound reproduction device and sound transmission and reproduction system.SOLUTION: In a sound transmission and reproduction system, a transmission method for a signal T for transmission including sound signals of a plurality of channels C-1 to C-n transmits in real time the signal T for transmission including a binaural first category sound signal F and a non-binaural second category sound signal S. The binaural first category sound signal F is a sound signal of a part of channels. The non-binaural second category sound signal S is a sound signal of the other channel and a signal that is easier to percept unnaturalness aurally with respect to positional displacement and movement than the first category sound signal F.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a signal transmission method, a signal generation method, a signal reproduction method, an audio signal processing program, an audio transmission device, an audio reproduction device, and an audio transmission and reproduction system for creating a stereophonic audio signal to be reproduced particularly through headphones or the like. [Background technology]

[0002] 2. Description of the Related Art Conventionally, there have been VR headphones and HMDs (Head Mounted Displays) (hereinafter, referred to as "HMDs, etc.") capable of playing back content such as movies, Virtual Reality (VR), and Augmented Reality (AR).

[0003] In order to create a wider sound field, these HMDs use a Head-Related Transfer Function (HRTF) that takes into account the direction from the listener to the sound source to reproduce binaural sound, a three-dimensional sound that localizes sound outside the listener's head. By measuring the direction of the listener's head relative to the sound source using an acceleration sensor, position sensor, etc. (head tracking), more accurate binaural sound, a two-channel (2ch) three-dimensional sound, can be generated (binauralization) and a more realistic sound image can be obtained.

[0004] Furthermore, in the past, in binaural conversion using HRTFs, the calculation of the actual audio signal often used head-related impulse responses (hereinafter referred to as "HRIRs"), which express head-related transfer functions on the time axis (hereinafter, HRTFs or HRIRs will be referred to simply as "HRTFs, etc.").

[0005] Patent Document 1 describes a technique for capturing, acoustically pre-processing, encoding, decoding, and binaural (binaural) rendering of directional audio of an audio scene. The technique of Patent Document 1 includes an apparatus adapted to modify directional characteristics of the captured directional audio in response to spatial data of a microphone system capturing the directional audio. The apparatus is configured to modify directional characteristics of the received directional audio by head tracking in response to the received spatial data. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent Publication No. 2021-5822 Summary of the Invention [Problem to be solved by the invention]

[0007] In conventional head tracking devices such as that described in Patent Document 1, directional information about the listener's head is transmitted, and the device that receives this information updates the HRTF, etc., which results in a delay in the update due to the round-trip communication time, making the sound image unnatural.

[0008] The present invention has been made in view of the above circumstances, and has an object to solve the above-mentioned problems. [Means for solving the problem]

[0009] The signal transmission method of the present invention is a method for transmitting a signal including audio signals of multiple channels, characterized in that the signal including a binauralized first category audio signal and a non-binauralized second category audio signal is transmitted in real time. The signal transmission method of the present invention is characterized in that the first category audio signal that has been binauralized is an audio signal of some channels, and the second category audio signal that has not been binauralized is an audio signal of the other channels. The signal transmission method of the present invention is characterized in that the second category audio signal is a signal that is more likely to be perceived as auditorily unnatural with respect to positional deviation and movement than the first category audio signal. The signal transmission method of the present invention is characterized in that it is determined whether the audio signals of the multiple channels are likely to be perceived as auditorily unnatural depending on the azimuth and elevation angle. The signal transmission method of the present invention is characterized in that the first category audio signal is a low-frequency component, and the second category audio signal is a high-frequency component. In the signal transmission method of the present invention, the first category audio signal is a representative point signal obtained by grouping a signal including audio signals of the multiple channels in a direction that is smaller than the number of channels of the signal. The signal generation method of the present invention is a signal generation method for generating a signal including audio signals of multiple channels, and is characterized in that it binauralizes only a first category audio signal based on received directional information of a listener, and generates the signal including the binauralized first category audio signal and a second category audio signal that is not binauralized. The signal reproduction method of the present invention is a method for reproducing a signal including audio signals of multiple channels, and is characterized in that it acquires a signal transmitted by the signal transmission method or a signal generated by the signal generation method, binauralizes the received second category audio signal, and generates a binaural signal by adding the binauralized second category audio signal and the first category audio signal that has already been binauralized when received. The signal reproduction method of the present invention is a signal reproduction method executed by an audio generation and reproduction system including an audio transmission device that generates and transmits a signal including audio signals of multiple channels, and an audio reproduction device that reproduces the signal transmitted from the audio transmission device, characterized in that the audio transmission device generates a representative point signal that groups the signal including the audio signals of the multiple channels in a direction fewer than the number of channels of the signal, transmits the representative point signal in real time, and the audio reproduction device binauralizes the representative point signal to generate a binaural signal. The audio signal processing program of the present invention is an audio signal processing program executed by an audio transmission device, and is characterized in that the audio transmission device binauralizes only a first category audio signal based on received directional information of a listener, and generates a signal including the binauralized first category audio signal and a second category audio signal that is not binauralized. The audio signal processing program of the present invention is an audio signal processing program executed by an audio playback device, and is characterized in that the audio playback device acquires a signal transmitted by the signal transmission method or a signal generated by the signal generation method, binauralizes the received second category audio signal, and generates a binaural signal by combining the binauralized second category audio signal with the first category audio signal that has already been binauralized when received. The audio transmission device of the present invention is an audio transmission device that generates a signal including audio signals of multiple channels, and is characterized in that it comprises a first binauralization unit that binauralizes only a first category audio signal based on directional information of a listener received from an audio playback device, and a signal generation unit that generates the signal including the first category audio signal binauralized by the first binauralization unit and a second category audio signal that is not binauralized. The audio playback device of the present invention is a playback device that plays a signal including audio signals of multiple channels, and is characterized in that it comprises a signal acquisition unit that acquires the signal including a binauralized first category audio signal and a second category audio signal that is not binauralized, a second binauralization unit that binauralizes the second category audio signal acquired by the signal acquisition unit, a binaural generation unit that generates a binaural signal by adding the second category audio signal binauralized by the second binauralization unit and the first category audio signal, and an audio output unit that outputs the binaural signal generated by the binaural generation unit. The audio transmission and playback system of the present invention is an audio generation and playback system including an audio transmission device that generates a signal including audio signals of multiple channels, and an audio playback device that plays the signal generated and transmitted by the audio transmission device, wherein the audio transmission device includes a first binauralization unit that binauralizes only a first category audio signal based on directional information of a listener received from the audio playback device, a signal generation unit that generates the signal including the first category audio signal binauralized by the first binauralization unit and a second category audio signal that is not binauralized, and a second category audio signal that is not binauralized. the audio playback device further comprises a signal acquisition unit that acquires the signal generated by the audio transmission device, a second binauralization unit that binauralizes the second category audio signal acquired by the signal acquisition unit, a binaural generation unit that generates a binaural signal by adding the second category audio signal binauralized by the second binauralization unit and the first category audio signal, an audio output unit that outputs the binaural signal generated by the binaural generation unit, and a direction transmission unit that transmits directional information of the listener to the audio transmission device. Effect of the Invention

[0010] According to the present invention, by transmitting a transmission signal in real time that includes a binauralized first category audio signal and a non-binauralized second category audio signal, it is possible to share the binauralization between the transmitting side and the receiving side, thereby providing a signal transmission method that can prevent the sound image from becoming unnatural. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram showing a configuration of an audio transmission and reproduction system according to an embodiment of the present invention. [Diagram 2] 3 is a conceptual diagram of an audio transmission and playback process according to the embodiment of the present invention. FIG. [Diagram 3] 4 is a flowchart of an audio transmission and playback process according to the embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] <Embodiment> [Control configuration of audio transmission and playback system X] First, with reference to FIG. 1, a control configuration of an audio transmission and reproduction system X according to an embodiment of the present invention will be described. The audio transmission and reproduction system X includes an audio transmission device 1 and an audio reproduction device 2.

[0013] The audio transmission device 1 is a device that encodes and transmits audio signals of multiple channels (hereinafter referred to as "source audio signals"). In addition, the audio transmission device 1 may be capable of generating the source audio signal itself by using or synthesizing the original audio signal. In this embodiment, the audio transmission device 1 is, for example, a parent terminal such as a smartphone, a mobile phone, an HMD, a gaming PC (Personal Computer) which may be portable, a consumer or store-use game machine, a server, a television, a television (video) conferencing system, remote conferencing equipment, an audio listening assistance device, a hearing aid, other home appliances, equipment in movie theaters or public viewing venues, a dedicated audio encoder, a switcher, a PA, other broadcasting and audio playback equipment, etc.

[0014] The audio reproduction device 2 is a device that reproduces the transmission signal T that is generated and transmitted by the audio transmission device. In this embodiment, the audio playback device 2 is, for example, an HMD (Head-Mounted Display) for XR (Extended Reality) such as VR (Virtual Reality), AR (Augmented Reality), MR (Mixed Reality), and SR (Substitutional Reality) that is capable of head tracking, an ear speaker, earphone, headphone, etc. (hereinafter referred to as "headphones, etc.") equipped with a head tracking sensor, an ear-hook type smart device, a stereophonic playback device to which headphones, etc. are connected, a dedicated game console, a content playback device, or other audio playback device.

[0015] In this embodiment, the audio transmission device 1 and the audio reproduction device 2 are connected wirelessly or wired. In the case of wireless, short-distance wireless including Bluetooth (registered trademark) and wireless LAN (Wireless Local Area Network, Wifi), Li-Fi and infrared communication, mobile phone networks such as 4G and 5G, medium-distance wireless communication, satellite radio communication networks, and other wireless communication technologies may be used. In the case of wired, the devices may be connected by various USB (Universal Serial Bus), wired LAN, wired WAN (Wide Area Network), HDMI (registered trademark), DP (DisplayPort), and other dedicated lines. Furthermore, the audio transmission device 1 and the audio reproduction device 2 may be connected by Bluetooth (registered trademark), TCP / IP, UDP, other IP networks, and other communication methods. In addition, whether to use a wireless or wired connection between the audio transmission device 1 and the audio reproduction device 2 may be selectable depending on the distance between the devices, convenience, etc. Furthermore, the audio transmission device 1 and the audio reproduction device 2 may be connectable by a plurality of methods.

[0016] In this embodiment, binaural audio is generated from an audio signal including channels C-1 to Cn as a source audio signal. Any one of the multiple channels C-1 to Cn will be simply referred to as "channel C" below.

[0017] Here, in the audio transmission and reproduction system X, the audio transmission device 1 according to the present embodiment divides source audio signals into a first category and a second category. Of the divided signals, only the first category signal is binauralized, and a binauralized signal (first category audio signal F) is generated. Then, a transmission signal T including the binauralized first category audio signal F and the non-binauralized second category audio signal S is transmitted to the audio reproduction device 2.

[0018] On the other hand, the audio reproduction device 2 according to the present embodiment acquires the transmission signal T transmitted by the audio transmission device 1, and performs control to binauralize the received second category audio signal S. The audio reproduction device 2 generates a binaural signal by adding up this binauralized second category audio signal S and the first category audio signal F that has already been binauralized when received. Then, the audio reproduction device 2 outputs this binaural signal.

[0019] That is, in this embodiment, the audio transmission device 1 and the audio reproduction device are controlled to transmit and receive (transmit) data such as the signal T for transmission. A delay of several tens of milliseconds to several hundreds of milliseconds or more may occur in the transmission between the audio transmitting device 1 and the audio reproducing device 2. Furthermore, the length of this delay may differ depending on whether the connection is wireless or wired.

[0020] Next, the functional configuration of each device in the audio transmission and reproduction system X will be described. The audio transmission device 1 according to this embodiment includes a binaural conversion unit 11 (first binaural conversion unit), a signal generation unit 12, and a transmission unit 13 as functional components. The audio reproduction device 2 includes a signal acquisition unit 20, a binaural conversion unit 21 (second binaural conversion unit), a binaural generation unit 22, an audio output unit 23, and a direction transmission unit 24.

[0021] The binaural conversion unit 11 and the binaural conversion unit 21 are calculation units that perform binaural conversion by convolving HRTF or the like with an audio signal. In this embodiment, the binauralization unit 11 binauralizes only the first category signals among the source audio signals based on the listener's direction information D received from the audio reproduction device 2, and generates the first category audio signal F. The binauralizer 21 binauralizes the second category audio signal S, which is a second category signal acquired by the signal acquirer 20.

[0022] In this embodiment, the binaural rendering unit 11 may determine whether or not the source audio signal is likely to be perceived as auditorily unnatural, based on directional information D of the received listener.

[0023] Here, the binaural conversion unit 11 may determine whether or not the azimuth and elevation angle are likely to cause auditory unnaturalness to be perceived for each channel C. In this case, the binaural conversion unit 11 may determine the azimuth and elevation angle for each channel C based on the direction information D of the listener and the direction information set for the channel C. Alternatively, the binaural unit 11 may determine that the low-frequency components of the channel C of the source audio signal are the first category audio signal F. The low-frequency components may be audio components of 1000 to 500 Hz or less divided into frequency bands, for example.

[0024] The signal generating unit 12 generates a transmission signal T including the first category audio signal F binauralized by the binauralizing unit 11 and the second category audio signal S that is not binauralized.

[0025] The transmitting unit 13 transmits in real time the transmission signal T generated by the signal generating unit 12. Specifically, the encoded transmission signal T is transmitted from the audio transmitting device 1 to the audio reproducing device 2 by the various wired or wireless methods described above. In addition, the transmission unit 13 can also receive directional information D of the listener acquired by the audio reproduction device 2.

[0026] The signal acquiring unit 20 acquires the transmission signal T generated by the audio transmitting device 1. The signal acquiring unit 20 also acquires the transmission signal T by the various wired or wireless methods described above.

[0027] The binaural generating unit 22 generates a binaural signal by combining the audio signals of the other channels binauralized by the binaural generating unit 21 with the audio signals of some of the channels.

[0028] The audio output unit 23 outputs the binaural signal generated by the binaural generation unit 22 as audio for both ears. In this embodiment, the audio output unit 23 includes, for example, a D / A converter and an amplifier for headphones, and reproduces audio through headphones such as an HMD. Furthermore, the audio output unit 23 may drive vibration elements for bass or a subwoofer. Alternatively, the audio output unit 23 may encode the audio signal and output it as an audio file or streaming audio for playback.

[0029] The direction transmitting unit 24 obtains directional information D of the listener regarding listening to the sound, and transmits it to the sound transmitting device 1. Specifically, the direction transmitting unit 24 may obtain the directional information D of the listener from a head tracking device such as an HMD, and transmit it to the sound transmitting device 1.

[0030] Here, in this embodiment, the source audio signal can be an audio signal from each direction of the content, an audio signal per content unit, an audio signal of a sound source object, an audio signal divided by frequency band, multiple audio signals of remote call participants, etc.

[0031] The content may be, for example, data of movies and music, games, audiobooks, data of electronic books capable of voice synthesis, broadcast data of television and radio, various voice data related to operation instructions of car navigation and various home appliances, XR content, and other data capable of voice output. Among these, movies also include musical instrument performances and lectures. Alternatively, background music and sound effects from games, MIDI files, voice call data of mobile phones and transceivers, and data of synthesized voice of text in messengers may be the content. In this case, channel C may include, as sound source objects, voice signals derived from objects such as vocal voices, musical instrument performances, vehicles, and game characters, and voice signals of people such as actors, narrators, storytellers, storytellers, and other speakers who are sound sources. Channel C may also be voice signals of remote participants in one-to-one, one-to-multiple, and multiple-to-multiple live events between locations, video conference systems, etc. These voice signals are set in a spatial arrangement relationship as directional information within the content. Alternatively, content can include background music and sound effects from games, music files such as MIDI files, voice call data from mobile phones or walkie-talkies, synthesized voice data for text in messengers, and other data that cannot be played directly.

[0032] Alternatively, when the source audio signal is an audio signal of a remote participant, it is possible to use an audio signal from a microphone at a live venue, an audio signal uttered by a user (participant) of various messengers or video conference application software (hereinafter simply referred to as "app") on a PC (Personal Computer) or smartphone, etc. Directional information added by a mixer, the direction of the participant's head in the camera, or the direction of an avatar placed in a virtual space, etc. may be added.

[0033] In either case, the source audio signal may be an audio signal recorded by a network or directly connected microphone. In this case, directional information may also be added to the audio signal. Alternatively, any combination of the above-mentioned contents and audio signals of remote participants may be used. Furthermore, the channel C may be set with directional information even if it is something other than an audio signal. Furthermore, in this embodiment, the source audio signal may be a "destination signal" for reproducing the direction for binauralization.

[0034] Here, in this embodiment, in the source audio signal, directional information may be set for each channel C. This directional information may be in an absolute coordinate system or a relative coordinate system with respect to the listener. In addition, the details and type of the above-mentioned content may also be set for each channel C.

[0035] In this embodiment, the transmission signal T may be, for example, a signal of various codecs capable of transmitting and receiving audio signals of multiple channels via Bluetooth (registered trademark) in the case of wireless communication.

[0036] In this embodiment, the first category audio signal F is an audio signal that is partially binauralized by the audio transmission device 1 from a source audio signal. Specifically, the first category sound signal F may be a sound signal of a part of the channels C-1 to Cn (hereinafter, referred to as "a part of the channels"). More specifically, the first category audio signal F may be a signal that is less likely (less likely to be) perceive as auditorily unnatural with respect to positional displacement and movement than the second category audio signal S. This signal may be a portion of the source audio signal for which a slow HRTF update speed is not a problem.

[0037] On the other hand, the second category audio signal S is an audio signal that is binauralized from a source audio signal by the audio reproduction device 2, but not by the audio transmission device 1. Specifically, the second category audio signal S may be an audio signal of a channel other than the certain channels. More specifically, the second category audio signal S may be a signal that is more likely to be perceptually unnatural with respect to displacement and movement than the first category audio signal F. This signal may be a portion of the source audio signal that requires a fast update rate of the HRTF.

[0038] [Hardware configuration of audio transmission device 1 and audio reproduction device 2] The audio transmission device 1 and the audio reproduction device 2 include, for example, various circuits, such as control means (control units) such as an ASIC (Application Specific Processor), a DSP (Digital Signal Processor), a CPU (Central Processing Unit), an MPU (Micro Processing Unit), and a GPU (Graphics Processing Unit).

[0039] Furthermore, the audio transmission device 1 and the audio reproduction device 2 may include, as a storage means (storage unit), a storage unit such as a semiconductor memory such as a ROM (Read Only Memory) or a RAM (Random Access Memory), a magnetic recording medium such as a HDD (Hard Disk Drive), or an optical recording medium. The ROM may include a flash memory or other writable or recordable recording medium. Furthermore, instead of the HDD, an SSD (Solid State Drive) may be provided. The storage unit may store the control program and various contents according to the present embodiment. Of these, the control program is a program for realizing each functional configuration and each method including the audio signal processing program of the present embodiment. The control program includes an embedded program such as firmware, an OS (Operating System), and an application.

[0040] The content in the storage unit according to this embodiment may be acquired by downloading a file or a chunk of data transmitted via a wired or wireless connection, or may be acquired stepwise by streaming or the like. Furthermore, the application according to the present embodiment may be an application such as a media player that plays back content, or an application for a messenger or video conference.

[0041] In addition, the audio transmission device 1 and the audio playback device 2 may be equipped with a direction calculation means including a GNSS (Global Navigation Satellite System) receiver that calculates the direction in which the listener is facing, an in-room position direction detector, an acceleration sensor, a gyro sensor, a geomagnetic sensor, etc. capable of head tracking, and a circuit that converts the output of these into direction information.

[0042] Furthermore, the audio transmitting device 1 and the audio reproducing device 2 may include a display unit such as a liquid crystal display, an organic EL display, or an LED (Light Emitting Diode), an input unit such as a button, a keyboard, a pointing device such as a mouse or a touch panel, and an interface unit for connecting to various devices wirelessly or via a wire. Among these, the interface unit may include an interface such as a flash memory medium such as a micro SD (registered trademark) card or a USB (Universal Serial Bus) memory, a LAN board, a wireless LAN board, a serial interface, a parallel interface, etc.

[0043] Moreover, the audio transmitting device 1 and the audio reproducing device 2 can realize each method according to the present embodiment using hardware resources by the control means executing various programs mainly stored in the storage means. These various programs include the audio signal processing program according to the present embodiment. Note that a part or any combination of the above-described configurations may be configured in terms of hardware or circuits using ICs, programmable logic, FPGAs (Field-Programmable Gate Arrays), or the like.

[0044] [Audio transmission and reproduction processing by audio reproduction device 2] Next, an audio transmission and playback process by the audio playback device 2 according to the embodiment of the present invention will be described with reference to FIGS.

[0045] First, an overview of the audio transmission and playback process according to this embodiment will be described with reference to FIG. In the audio transmission and reproduction process according to this embodiment, audio signals (source audio signals) of multiple channels are divided into a first category audio signal F, for which a slow HRTF update speed is not a problem, and a second category audio signal S, for which a fast HRTF update speed is required. The first category audio signal F is then binauralized by the audio transmission device 1. Meanwhile, the second category audio signal S is binauralized by the audio reproduction device 2. As a result, by updating the second category signal S, which requires a fast update speed, after receiving communication, there is no delay in updating the head related transfer function according to head movement for the second category signal S, even if there is a delay in communication time, and it is possible to prevent the sound image from becoming unnatural.

[0046] In the audio transmission and playback processing of this embodiment, mainly in the audio transmission device 1 and the audio playback device 2, the control means cooperates with each part to control and execute a control program stored in a storage means using hardware resources, or processes it directly in each circuit. The details of the audio transmission and playback process will be described below step by step with reference to the flowchart of FIG.

[0047] (Step S101) First, the binaural conversion unit 11 of the audio transmission device 1 performs audio separation processing. The binaural unit 11 judges whether or not the channel C is likely to be perceived as auditorily unnatural when the update of the head-related transfer function in response to head movement is delayed from the source audio signal. Then, the binaural unit 11 separates the channel C in which the auditory unnaturalness is not likely (is difficult to be perceived) as a first category audio signal F. In this embodiment, the binaural conversion unit 11 separates the source audio signal into a portion for which the HRTF update speed must be fast and a portion for which a slower update speed does not pose a problem.

[0048] Specifically, the binauralization unit 11 receives directional information D of the listener from the audio reproduction device 2 via the transmission unit 13. Furthermore, the binauralization unit 11 can obtain the direction of the audio in the virtual space for each channel C. Then, based on the direction information D of the listener, the binaural conversion unit 11 calculates the channel C and the direction (azimuth and elevation angle) of the listener in the spatial arrangement including the virtual space.

[0049] Here, the binaural conversion unit 11 judges whether or not the azimuth and elevation angle are likely to cause the sound to be perceived as unnatural. As a specific example, the binauralization unit 11 may separate a channel C signal whose absolute values ​​of the azimuth and elevation angle with respect to the listener are equal to or greater than a specific angle threshold as a first category audio signal F. This angle threshold can be set separately for each of the azimuth and elevation angle based on experimental values, etc. On the other hand, the binauralization unit 11 may separate a channel C signal whose absolute values ​​of the azimuth and elevation angle are less than a specific angle threshold as a second category audio signal S.

[0050] In this example, the binauralizer 11 determines that the signal of channel C, which is upward, downward, or lateral to the listener, is not likely (is not likely) to be perceived as auditorily unnatural due to positional shifts and movements. For this reason, the binauralizer 11 separates the signal of channel C from the first category audio signal F and binauralizes it before transmission by the audio transmission device 1. Conversely, when the direction to the listener is close to the front, the binaural conversion unit 11 determines that the positional shift and movement are likely to be perceived as unnatural from an auditory perspective. For this reason, the binaural conversion unit 11 separates the signal of this channel C as a second category audio signal S.

[0051] For example, the binaural unit 11 may separate the channel C of a musical instrument performance as the first category audio signal F. This is because musical instrument sounds have a spread in components coming from the sides, and therefore it is expected that tracking by head tracking can be performed relatively slowly. On the other hand, the binauralization unit 11 may separate the vocal channel C as a second category audio signal S. This is because vocals are often located in the horizontal direction in front, and therefore the auditory sense is sensitive to the positional deviation or movement of the vocals. In other words, by performing binauralization using the audio reproduction device 2, it becomes possible to realize high-speed tracking of the head movement of the listener. Here, the binaural rendering unit 11 may determine the direction in the virtual space of channel C according to the details or type of the content of channel C, or may determine the direction according to setting information of channel C.

[0052] Alternatively, the binauralization unit 11 may determine and separate the low-frequency components as the first category audio signal F for each channel C of the source audio signal. In this case, the binauralization unit 11 may determine and separate the high-frequency components as the second category audio signal S. In other words, the binauralization unit 11 may separate the low-frequency components as the first category audio signal F and the high-frequency components as the second category audio signal S.

[0053] In this case, the binauralization unit 11 may separate each channel C into a channel for low-frequency components and a channel for high-frequency components using a band-pass filter such as a low-pass filter or a high-pass filter. The binauralization unit 11 may then determine the channel for low-frequency components as a first category audio signal F. The binauralization unit 11 may also determine the channel for high-frequency components as a second category audio signal S. Alternatively, the binaural conversion unit 11 may calculate the ratio of low-frequency components to high-frequency components for each channel C, and determine whether the channel is a first category audio signal F or a second category audio signal S based on this ratio.

[0054] In this example, the binauralization unit 11 may separate the low-frequency components as a first category audio signal F to be binauralized by the audio transmission device 1, since position shifts and movements are less likely to be perceived as auditory unnaturalness. On the other hand, since the high-frequency components are easily perceived as being unnatural due to positional shifts and movements, the binauralizer 11 may separate the high-frequency components as a second category audio signal S and binauralize them by the audio reproduction device 2. In this case, since the high-frequency components have little energy, the bit rate can be kept low even when encoded in multiple channels. On the other hand, since the high-frequency components play an important role in directional perception, binauralizing them by the audio reproduction device 2 makes it possible to quickly follow the head movement of the listener.

[0055] (Step S102) Next, the binaural conversion unit 11 performs a first binaural conversion process. Here, the binauralization unit 11 binauralizes (converts to two channels, left and right) each channel of the first category audio signal F by convolving HRTF and the like based on sound source direction information and listener direction information D. That is, the binauralization unit 11 binauralizes the first category audio signal F that is determined to be not easily (hard to be) perceived as being unnatural auditorily even if there is a delay due to transmission or the like. In the above example, many instrument sounds are separated as the first category audio signal F and binauralized (converted to two channels), making it possible to save transmission bandwidth during encoding.

[0056] On the other hand, the binauralization unit 11 does not binauralize the second category audio signal S that is determined to be easily perceivably unnatural from an auditory perspective. That is, since the second category audio signal S is an audio signal that is easily perceivably unnatural from an auditory perspective due to delay or the like, it is binauralized by the binauralization unit 21 provided in the audio reproduction device 2 that is closer to the listener. That is, this second category audio signal S is binauralized by the audio reproduction device 2 after transmission, which will be described later.

[0057] (Step S103) Next, the signal generating unit 12 performs a process of generating a signal for transmission. The signal generating unit 12 generates a transmission signal T including a first category audio signal F that has already been binauralized and a second category audio signal S that has not been binauralized. For example, the signal generating unit 12 may encode the first category audio signal F and the second category audio signal S with various codecs such as AAC, AptX LL, HD, LDAC, and LHDC when transmitting a signal by Bluetooth (registered trademark), and include the encoded signal as the transmission signal T. Furthermore, the signal generating unit 12 may encode audio signals of multiple channels, such as 2 channels (2ch), 5.1 channels (5.1ch), 7.1 channels (7.1ch), and 22.2 channels (22.2ch), in accordance with audio encoding methods such as MPEG-2 AAC, MPEG-4 AAC, MP3, Dolby (registered trademark) Digital, and DTS (registered trademark), as the transmission signal T. In addition, the signal generating unit 12 may bundle the audio signals of the multiple channels and control signals (control signals) into packets, etc., to generate the transmission signal T.

[0058] (Step S104) Next, the transmission unit 13 performs a transmission process for the signal to be transmitted. The transmission unit 13 transmits the transmission signal T in real time. At this time, the transmission unit 13 transmits the transmission signal T by various wired or wireless methods. This makes it possible to perform a "hybrid" transmission of the binauralized first category sound signal F and the non-binauralized second category sound signal S.

[0059] (Step S201) Here, the processing of the audio reproduction device 2 will be described. The signal acquisition unit 20 of the audio reproduction device 2 performs signal acquisition processing. The signal acquisition unit 20 receives (acquires) the transmission signal T generated by the voice transmission device 1, and temporarily stores it in a storage means.

[0060] (Step S202) Next, the binaural conversion unit 21 performs a second binaural conversion process. The binauralizer 21 binauralizes the second category audio signal S. As described above, the second category audio signal S is an audio signal that is likely to be perceived as auditorily unnatural.

[0061] (Step S203) Next, the binaural generation unit 22 performs a binaural signal generation process. The binaural generating unit 22 generates a binaural signal by combining the audio signals of the other channels binauralized by the binaural generating unit 21 with the audio signals of some of the channels. In this embodiment, the binaural signal may include an L signal for the left ear of the listener and an R signal for the right ear of the listener. Furthermore, the binaural signal may be a digital signal that can be heard by the listener by decoding digital data and playing it back at the audio output unit 23.

[0062] (Step S204) Next, the audio output unit 23 performs audio output processing. The audio output unit 23 outputs the audio signal generated by the binaural generation unit 22. This output may be, for example, a 2-channel analog audio signal corresponding to the left and right ears of the listener. This allows the audio output unit 23 to reproduce an audio signal corresponding to a virtual sound field as a 2-channel audio signal through headphones. This allows the listener to hear left and right sounds in a stereophonic environment.

[0063] (Step S205) Next, the direction transmitting unit 24 performs a direction transmitting process. The direction transmitting unit 24 can acquire the head direction of the listener from head tracking using a gyro sensor or the like of an HMD or a smartphone, direction information such as the orientation of an avatar in a virtual space, and the like. The direction transmitting unit 24 transmits the acquired direction information D of the listener to the audio transmitting device 1. At this time, the direction transmitting unit 24 may directly transmit the direction information D of the listener via a wired or wireless connection, separately from the transmission signal T. Alternatively, the direction transmitting unit 24 may be configured to transmit the direction information D of the listener via the signal acquiring unit 20, via the same wireless or wired connection as the transmission signal T, such as Bluetooth (registered trademark). This completes the audio transmission and playback process according to this embodiment.

[0064] With the above configuration, the following effects can be obtained. In recent years, it has become common to connect a smartphone to an HMD or headphones to enjoy virtual sound fields (3D sound) for VR or AR. However, in the past, in order to generate binaural signals for 3D sound playback through headphones, the transmitting side of a smartphone or other device performed convolution processing of the HRIRs of the sound source direction corresponding to each of the multiple sound source signals. In other words, out-of-head localization was realized and 3D sound was generated by convolving the head-related transfer function of the sound source direction with the sound signal of each channel of each source sound signal to generate binaural signals.

[0065] In this case, in a head tracking system, the relative direction between the sound source and the listener changes when the listener moves his / her head, so the listener's head direction is detected, this direction information is sent to a smartphone, etc., and the HRIR is updated accordingly. In other words, a stationary sound source is generated by calculating the direction of the HRTF to be used from the relationship between the listener's head direction and the sound source position, and instantly updating it. The same processing is performed for moving sound sources, and a moving sound source is generated according to the listener's head movement.

[0066] However, delays in communication between smartphones and headphones have been an issue. Specifically, in wireless headphones using Bluetooth (registered trademark) and other technologies, which have significant delays, it takes a total of about 200 milliseconds to transmit directional information detected by head tracking to a smartphone and then to transmit information such as HRTF updated based on that information to the headphones. In other words, there is a delay of 200 milliseconds or more in updating the HRTF relative to head movement.

[0067] For this reason, conventional headphones with head tracking had a problem in that the sound image was not stable in response to the listener's head movements because they were poor at tracking the listener's head movements. In other words, there was a delay in updating HRTF, etc., due to the round-trip communication time, and the sound image became unnatural when the listener moved their head. Specifically, when the listener rotated their head, the sound image of a stationary sound source would temporarily follow the head rotation. In fact, this problem has been observed in consumer earphones with head tracking. If the delay is not less than about 60 milliseconds, the unnaturalness of the sound image positioning is significantly perceived.

[0068] In contrast, (A) a signal transmission method according to an embodiment of the present invention is a method for transmitting a transmission signal T including audio signals of multiple channels, and is characterized in that the transmission signal T including a binauralized first category audio signal F and a non-binauralized second category audio signal S is transmitted in real time.

[0069] With this configuration, a signal transmission method is provided for transmitting an audio signal including three or more channels C, in which a part of the signal group is binauralized as a first category and the rest is transmitted as a second category without binauralization. By dividing the binauralization between the transmitting side and the receiving side, the second category audio signal can be binauralized at the listener's side, and the unnaturalness of the sound image can be suppressed. In other words, the effect of transmission delay can be minimized, making it difficult for the listener to perceive unnaturalness. This makes it possible to apply it to various XR application software such as games, movies, and home TV.

[0070] Conventionally, binauralization of multi-channel audio signals for headphone playback was performed by a transmitting device such as a smartphone. In this case, the audio signal to be transmitted is a 2ch binaural signal, so the deterioration of sound quality due to encoding for transmission was minimized. However, as described above, this caused a problem of unnatural sound images due to delays in sound image tracking caused by transmission delays.

[0071] On the other hand, if binaural processing is performed on the receiving side, such as headphones, rather than on the transmitting side to eliminate the unnaturalness of the sound image caused by such delays, delays in updating the HRTF due to communication do not occur, and the HRTF can be updated instantly according to the listener's head direction. However, in such a case, it is necessary to transmit the multi-channel audio signals to the receiving side as they are before converting them into binaural signals. This increases the transmission rate (transmission capacity), making it necessary to perform highly efficient coding with a high compression rate. This can cause sound quality degradation due to coding distortion.

[0072] In contrast, the signal transmission method according to the embodiment of the present invention can instantaneously track head movements and transmit signals without degradation of coding, thereby improving the signal-to-noise ratio (hereinafter, referred to as "SNR") of audio. Therefore, by applying the signal transmission method according to the present embodiment to a 3D sound playback system, such as a wireless headphone playback system connected to a smartphone, it becomes possible to play back sound with higher quality than before. In addition, by applying the method to smartphones and home appliances, it becomes possible to generate high-quality 3D sound from multi-channel audio signals, which can be applied to international standardization, etc.

[0073] (B) In a signal transmission method according to an embodiment of the present invention, the binauralized first category audio signal F is an audio signal of some channels, and the audio signal that is not binauralized is an audio signal of the other channels, which is a signal transmission method described in (A).

[0074] In this configuration, in signal transmission related to an audio signal including three or more channels C, the first category channel C is binauralized before transmission and then transmitted, and the second category channel C is binauralized after reception, and these are added together at the receiving side to generate a binaural signal. That is, the source audio signal may be divided into two categories, the first category audio signal F and the second category audio signal S, in units of channel C, and dynamic bit allocation may be performed between these two categories. That is, the allocation of the transmission band may be dynamically changed. In this way, by performing hybrid transmission in which the binauralization is shared between the transmitting side and the receiving side, it is possible to transmit an audio signal with high sound quality while suppressing unnaturalness of the sound field due to transmission delay.

[0075] In this case, for example, a configuration can be realized in which a channel C belonging to a first category audio signal F is binauralized by an audio transmission device 1 before encoding, and a channel C belonging to a second category audio signal S is encoded on a channel-by-channel basis and then binauralized after being decoded by an audio reproduction device 2.

[0076] (C) In a signal transmission method according to an embodiment of the present invention, the second category audio signal S is a signal that is more likely to be perceived as auditorily unnatural in response to position shifts and movement than the first category audio signal F, and is characterized by being a signal transmission method described in (A) or (B).

[0077] With this configuration, audio signals of multiple channels are divided into first category audio signals F, for which a slow HRTF update speed is not a problem because unnaturalness due to a positional shift in the sound source direction is not easily (hardly) perceived auditorily, and second category audio signals S, for which a fast HRTF update speed is required because unnaturalness due to a positional shift in the sound source direction is easily perceived auditorily. Then, only the first category audio signals F, for which unnaturalness is not easily (hardly) perceived auditorily, are binauralized by the audio transmission device 1, and the second category audio signals S, for which unnaturalness is easily perceived auditorily, are binauralized by the audio reproduction device 2. This reduces the number of channels in the transmitted signal, minimizing coding distortion, while enabling high-speed HRTF updates for necessary sound sources, resulting in high-quality, stable sound field reproduction even when head movement is present.

[0078] (D) A signal transmission method according to an embodiment of the present invention is characterized in that the source audio signal, which is an audio signal of multiple channels, is determined as to whether it is likely to be perceived as auditory unnatural based on the azimuth and elevation angle.

[0079] With this configuration, it is possible to group each channel C into two categories according to the azimuth angle and elevation angle. For this reason, for example, a musical instrument sound having a wide range of components coming from the sides can be determined as a first category audio signal F that is less likely to be perceived as unnatural to the auditory sense because the tracking may be relatively slow. For this reason, the signal capacity of the musical instrument sound can be reduced by binauralizing the musical instrument sound using the audio transmission device 1. In other words, by binauralizing many musical instrument sounds and leaving them only in left and right channels (2ch), the capacity required for encoding can be significantly reduced.

[0080] On the other hand, for example, vocals, which are often located in the horizontal direction in front, are easily perceived as unnatural to the auditory sense, and can be determined to be a second category audio signal S that is sensitive to positional shifts. For this reason, only the second category audio signal S of the vocals is transmitted as an independent signal, and the first category audio signal F of the musical instrument performance is binauralized in advance and transmitted. Then, by binauralizing the second category audio signal S by the audio reproduction device 2, high-speed tracking of head movement can be achieved.

[0081] (E) In a signal transmission method according to an embodiment of the present invention, the first category audio signal F is a low-frequency component and the second category audio signal S is a high-frequency component, which is a signal transmission method described in (C).

[0082] By configuring in this way, grouping can be performed for each channel C into two categories according to the transmission bandwidth. Therefore, for example, it is possible to binauralize the low-frequency components before transmission and then transmit them, binauralize the high-frequency components after reception, and add them together at the receiving side to generate a binaural signal. Specifically, it is possible to binauralize the low-frequency components of the instrument sound before transmission, and encode only the high-frequency components in an independent channel and binauralize them at the receiving side. In this case, since the high-frequency components of the audio signal have little energy, even if they are encoded for many channels, the bit rate (transmission bandwidth) can be kept low. On the other hand, since the high-frequency components are important in the listener's perception of direction, they are preferable because they need to follow the head movement at high speed.

[0083] (F) A signal generation method according to an embodiment of the present invention is an audio generation method for generating a transmission signal T including audio signals of multiple channels, and is characterized in that it binauralizes only a first category audio signal F based on received directional information D of the listener, and generates a transmission signal T including the first category audio signal F binauralized by the binauralization unit 11 and a second category audio signal S that has not been binauralized.

[0084] With this configuration, in signal transmission of an audio signal including three or more channels C, the audio transmission device 1, such as a smartphone, can binauralize only the first category audio signal F using directional information D of the listener, which may be delayed.

[0085] (G) A signal reproduction method according to an embodiment of the present invention is a reproduction method for a transmission signal T including audio signals of multiple channels, characterized in that it acquires a transmission signal T transmitted by a signal transmission method described in any one of (A) to (E) or a transmission signal T generated by a signal generation method described in (F), binauralizes the received second category audio signal S, and generates a binaural signal by combining the binauralized second category audio signal S with a first category audio signal F that has already been binauralized when received.

[0086] With this configuration, in signal reproduction related to an audio signal including three or more channels, some channels C of the signal group are binauralized before transmission in communication, and other channels C are binauralized after reception in communication, and the binaural signals can be generated by adding them together. That is, in signal reproduction related to a source audio signal, a part of the received transmission signal T can be received as it is, and a part can be binauralized after reception, and the binaural signals can be generated by adding them together. Therefore, for source audio signals such as content, one-to-one connection, one-to-multipoint connection, multipoint-to-multipoint connection, and remote conference, it is possible to realize sound field reproduction that does not make the listener perceive unnaturalness while minimizing the coding distortion of the transmission signal T. In addition, the audio signal of the binaural signal can be outputted through headphones, HMD, etc., to experience realistic sound.

[0087] (H) An audio transmission and reproduction system X according to an embodiment of the present invention is a reproduction system including an audio transmission device 1 that generates a transmission signal T including audio signals of multiple channels, and an audio reproduction device 2 that reproduces the transmission signal T generated by the audio transmission device. The audio transmission device 1 includes a binauralization unit 11 that binauralizes only a first category audio signal F based on directional information D of a listener received from the audio reproduction device 2, a signal generation unit 12 that generates a transmission signal T including the first category audio signal F binauralized by the binauralization unit 11 and a second category audio signal S that is not binauralized, and a signal generation unit 12 that generates a transmission signal T including the first category audio signal F binauralized by the binauralization unit 11 and a second category audio signal S that is not binauralized. The audio reproduction device 2 is characterized by comprising a signal acquisition unit 20 that acquires the transmission signal T generated by the audio transmission device 1, a binauralization unit 21 that binauralizes the second category audio signal S acquired by the signal acquisition unit 20, a binaural generation unit 22 that generates a binaural signal by adding the second category audio signal S binauralized by the binauralization unit 21 and the second category audio signal F, an audio output unit 23 that outputs the audio signal generated by the binaural generation unit 22, and a direction transmission unit 24 that transmits direction information D of the listener to the audio transmission device 1.

[0088] With this configuration, it is possible to transmit the signal with minimal coding distortion while minimizing the effects of transmission delays, making it difficult for listeners to perceive unnaturalness.

[0089] Other Embodiments In the above-described embodiment, an example has been described in which each of the channels C-1 to Cn of the source audio signal is directly divided into a first category audio signal F and a second category audio signal S on a channel C basis, and an example in which the signals are divided into either low-frequency components or high-frequency components. However, the first category audio signal may be obtained by clustering or panning the channels C-1 to Cn of the source audio signal, which are several to several hundreds (n) of signals present in a 3D (three-dimensional) space, in specific m directions, which is less than the number of channels n. In this case, the signals in the m directions may be generated in the audio transmission device 1 by calculating the sum of the signals that are time-shifted after applying an appropriate gain to each of the channels C-1 to Cn, and may be compiled as a representative point signal present at the position of a representative point in the m directions. Then, this compiled representative point signal may be transmitted to the audio reproduction device 2. That is, in this example, the signals in the m directions (representative point signals) may be treated as the first category audio signal. In addition, the signals in the m directions (representative point signals) may be binauralized by convolving HRIR or the like in the audio reproduction device 2. In addition, in this case, the second category audio signal that is not binauralized may be transmitted together with the first category audio signal as a signal for transmission. Alternatively, the second category audio signal in this case may be a control signal for encoding the audio signal, or the like.

[0090] In this case, as in the above-described embodiment, a delay may occur in the transmission between the audio transmitting device 1 and the audio reproducing device 2. The audio reproducing device 2 that receives the signal for transmission may combine (binauralize) the m signals while updating the HRIR in real time according to head direction information and the orientations of the m signals in accordance with the movement of the listener's head. At this time, the audio transmitting device 1 may take the sum of the signals distributed to the representative point of the channel C to be combined at the representative point, generate a sum signal, and transmit this sum signal. Then, the binauralization unit 21 of the audio reproducing device 2 may convolute the HRIR at the position of the representative point (HRIR in the direction of the representative point) with this sum signal to binauralize it and generate a binaural signal.

[0091] That is, in a signal transmission method according to another embodiment of the present invention, the first category audio signal may be a representative point signal in which a signal (source audio signal) including audio signals of multiple channels is grouped in a direction that is smaller than the number of channels of the source audio signal. Alternatively, in a signal reproduction method according to another embodiment of the present invention, an audio transmission device 1 may generate a representative point signal by consolidating a source audio signal in a direction that has fewer channels than the number of channels of the source audio signal, and transmit the representative point signal in real time. An audio reproduction device 2 may then binauralize the representative point signal to generate a binaural signal.

[0092] With this configuration, it is possible to consolidate several to several hundred (n) signals that need to be convolved with HRIR in XR or the like into signals in m directions by the audio transmission device 1, and then to binauralize them after transmitting them to the audio reproduction device 2. By consolidating them into m signals, the transmission band can be suppressed, and transmission can be easily performed wirelessly. In addition, the amount of calculation can be reduced compared to the conventional method of convolving HRIR into each sound source signal individually. In addition, by converting the sound to binaural using the sound reproduction device 2, it is possible to control the orientation of the virtual speaker, which is the representative direction of panning, in conjunction with the listener's head movement. This allows the sound source position to be updated instantly in response to head movement without being affected by delays due to transmission. Therefore, the HRIR can be updated instantly in response to head movement, and delays become no longer an issue. In other words, by reducing the number of sound source objects to be reproduced by panning, wireless transmission from a smartphone or the like to headphones or the like becomes possible. This makes it possible to realize an HMD, wireless headphones, etc. with an extremely high sense of realism.

[0093] In the above embodiment, the angular directions of the listener in the up, down, left and right directions are taken into consideration as directional information. However, it may be configured to take into account only the left-right direction, or to take into account 6DOF (Six Degrees of Freedom).

[0094] In the above-described embodiment, the judgment as to whether or not the sound is likely to be perceived as auditorily unnatural is based on either the azimuth and elevation angle or the low-frequency components and high-frequency components. However, in the binaural conversion section 11, the judgment based on the azimuth and elevation angle and the judgment based on the low-frequency components and high-frequency components may be combined. For example, a vocal channel C may be designated as a second category audio signal S, while the low-frequency components of an instrumental sound channel C may be designated as a first category audio signal F, and the high-frequency components may be separated into a channel separate from the vocal channel C as a second category audio signal S.

[0095] In the above embodiment, in separating the first category audio signal and the second category audio signal, bit allocation, i.e., allocation of a transmission band, is performed according to the energy of each component. However, the band of the signal for transmission may be different between the first category audio signal and the second category audio signal. For example, the band of the first category audio signal may be narrower than that of the second category audio signal. In this case, the band of the high frequency components may be set according to the amount of energy as in the above embodiment, while only the band of the low frequency components may be reduced. Alternatively, the ratio of the bands of the first category audio signal and the second category audio signal may be changed depending on the moving speed or acceleration of the directional information D of the listener. Alternatively, the ratio of the bandwidth between the first category audio signal and the second category audio signal may be changed depending on whether the connection is wireless or wired, or simply depending on the magnitude of the delay due to transmission. In this case, if the delay is less than several tens of milliseconds due to a high-speed wired connection or the like, the audio signal may not be divided into the first category audio signal and the second category audio signal.

[0096] Such a configuration makes it possible to conserve the transmission band while preventing the listener from noticing the positional deviation of the sound source direction, thereby enabling high-quality audio transmission.

[0097] In the above embodiment, it has been described that the HRIR is convolved with the audio signal of channel C in binaural conversion. However, a similar process can be achieved by transforming the audio signal of channel C into the frequency domain and applying the HRTFs. Furthermore, it is also possible to configure the system so that the low frequency components and the high frequency components are separated and the HRTF is applied only in the case of hybrid transmission. Specifically, as in the above-mentioned embodiment, by using the HRTFs for the low and high frequency ranges based on the frequency band near or slightly higher than the frequency band where the human hearing sensitivity is high, more accurate synthesis is possible.

[0098] Further, although the above-described embodiment did not describe the differences between HRTFs, etc., in practice, the binaural conversion section 21 may be able to select the user's personal HRIR or an HRIR generated by an HRIR database from the HRIR table. Furthermore, when the speaker and listener are transformed into avatars in a virtual space, the binauralizer 21 can select HRIRs accordingly. That is, for example, in the case of an avatar with ears pointing upwards, such as a cat or rabbit avatar, it is possible to select an HRIR that matches the avatar.

[0099] Furthermore, the binaural unit 21 can also superimpose the direct sound of channel C and the reflected sound from the environment by a separate process such as convolution, thereby further enhancing the sense of reality. With this configuration, it is possible to reproduce clearer, more realistic sounds.

[0100] In addition, in the above embodiment, an example in which the audio output unit 23 reproduces audio in two channels, left and right, has been described. Regarding this, it is also possible to configure so that the multiple channels can be played back using headphones or the like that can play back the multiple channels as separate channels. Furthermore, the binaural generation unit 22 may be capable of separately generating signals for a subwoofer or vibration and playing these.

[0101] In the above embodiment, the audio reproducing device 2 is described as being integrally configured. However, the audio reproducing device 2 may be configured so that a device for receiving audio is connected to a device for actually reproducing audio, such as a headset, headphones, or left / right separated earphones. That is, the audio reproducing device 2 itself may be configured with a plurality of devices in addition to the audio transmitting device 1. In addition, the audio transmitting device 1 itself may be configured with a plurality of devices. Furthermore, in the above embodiment, an example has been described in which the distance between the audio transmission device 1 and the audio reproduction device 2 is relatively short, but this distance may be long. In this case, the functions of the audio transmission device 1 can be executed by a server on an intranet or the Internet.

[0102] It is also possible to configure the audio playback device 2 to play back content stored on an optical medium or a flash memory card, generated sound source objects, etc. as source audio signals, and at this time transmit a first category audio signal F from this source audio signal to the audio transmission device 1 to convert it into binaural audio.

[0103] In the above embodiment, the audio reproduction device 2 is described as including the binaural generation unit 22 and the audio output unit 23. However, a configuration that does not include the binaural generation unit 22 and the audio output unit 23 is also possible. In this case, for example, data of audio signals such as generated binaural signals may be stored in a recording medium. In this case, the other audio reproduction device is applicable to all devices that can obtain the direction of channel C in a virtual space, such as devices equipped with a television or display, videophone through a display, videoconferencing, telepresence, etc.

[0104] In the above embodiment, an example in which directional information is added to the audio signal of channel C has been described. In this regard, in a situation where the speaker and listener are constantly changing during a conversation, such as the remote conference described above, directional information does not need to be added to the voice signal of channel C. In other words, when the current listener is the speaker, the direction of the speaker (current listener) can be estimated using the spoken voice signal, and this can be used as the direction of the listener from the perspective of the current speaker.

[0105] In this case, the direction transmitting unit 24 calculates the arrival direction of the audio signal of the L (left) channel signal (hereinafter referred to as the "L signal") and the R (right) channel signal (hereinafter referred to as the "R signal") as seen by the listener. At this time, the direction transmitting unit 24 may obtain the intensity ratio of the L channel and the R channel. It is also possible to estimate the arrival direction of the signal of each frequency component from the intensity ratio.

[0106] Alternatively, the direction transmitting unit 24 may estimate the direction of arrival of the audio signal from the relationship between the ITD (Interaural Time Difference) of signals of each frequency in HRTF etc. and the direction of arrival. The direction transmitting unit 24 may refer to the relationship between the ITD and the direction of arrival stored in the storage unit as a database.

[0107] Alternatively, it is possible to estimate the direction of a speaker or listener by performing face recognition from face image data of a person such as a speaker or listener in a content or video conference. That is, it is possible to estimate the direction even in a configuration without head tracking. Similarly, it may be possible to grasp the position of a speaker or listener in a space. Such a configuration makes it possible to support various flexible configurations. In addition, in an XR in which the sound source position is set in advance, it may be possible to obtain the direction of channel C from the positional relationship between channel C and listener L without estimating the sound source direction.

[0108] It goes without saying that the configurations and operations of the above-described embodiment are merely examples and can be modified as appropriate without departing from the spirit of the present embodiment. [Industrial Applicability]

[0109] The audio transmission device of this embodiment can provide an audio transmission and reproduction system that prevents the sound image from becoming unnatural when generating stereophonic sound, and can be used industrially. [Explanation of symbols]

[0110] 1. Audio transmission equipment 2 Audio playback device 11, 21 Binaural conversion section 12 Signal Generation Unit 13 Transmission section 20 Signal Acquisition Unit 22 Binaural generation part 23 Audio output section 24 Directional transmitter D. Directional information of listeners F First category speech signal S Second category speech signal T Transmission signal C-1~Cn Channel X Audio transmission and playback system

Claims

1. A method for transmitting a signal including a plurality of channel audio signals, comprising: Transmitting said signals in real time, said signals including a first category of binaurally processed audio signals and a second category of non-binaurally processed audio signals. A signal transmission method comprising:

2. The first category audio signals that have been binauralized are audio signals of some channels, and the second category audio signals that have not been binauralized are audio signals of other channels.

2. The signal transmission method according to claim 1 .

3. The second category audio signal is a signal that is more likely to be perceived as being auditorily unnatural with respect to position shift and movement than the first category audio signal.

2. The signal transmission method according to claim 1 .

4. The audio signals of the multiple channels are The degree to which the auditory unnaturalness is perceived is determined by the azimuth and elevation angle.

4. The signal transmission method according to claim 3.

5. The first category audio signal is a low-frequency component, and the second category audio signal is a high-frequency component.

4. The signal transmission method according to claim 3.

6. The first category audio signal is The signal including the audio signals of the plurality of channels is a representative point signal that is grouped in a direction that is smaller than the number of channels of the signal.

2. The signal transmission method according to claim 1 .

7. 1. A method for generating a signal including a plurality of channel audio signals, comprising: Based on the received listener's direction information, only the first category audio signal is binauralized; generating a signal including a first category audio signal that has been binauralized and a second category audio signal that has not been binauralized; A signal generating method comprising:

8. A method for reproducing a signal including a plurality of channel audio signals, comprising the steps of: A signal transmitted by the signal transmission method according to any one of claims 1 to 6 or a signal generated by the signal generation method according to claim 6 is acquired, binauralizing the received second category audio signal; The binauralized second category audio signal is summed with the first category audio signal that has already been binauralized when received to generate a binaural signal.

23. A signal reproducing method comprising:

9. A signal reproduction method executed by an audio generation and reproduction system including an audio transmission device that generates and transmits a signal including audio signals of a plurality of channels, and an audio reproduction device that reproduces the signal transmitted from the audio transmission device, comprising: The audio transmission device generates a representative point signal that is a combination of the signals including the audio signals of the plurality of channels in a direction smaller than the number of channels of the signals, and transmits the representative point signal in real time; The representative point signal is binauralized by an audio reproduction device to generate a binaural signal.

23. A signal reproducing method comprising:

10. A sound signal processing program executed by a sound transmission device, the sound transmission device Based on the received listener's direction information, only the first category audio signal is binauralized; A signal is generated that includes a first category audio signal that has been binauralized and a second category audio signal that has not been binauralized.

2. An audio signal processing program comprising:

11. An audio signal processing program executed by an audio playback device, the audio playback device acquiring a signal transmitted by the signal transmission method according to any one of claims 1 to 6 or a signal generated by the signal generation method according to claim 6; binauralizing the received second category audio signal; summing the binauralized second category audio signal with the first category audio signal that was already binauralized when received to generate a binaural signal; 2. An audio signal processing program comprising:

12. 1. An audio transmission device for generating a signal including audio signals of multiple channels, comprising: a first binauralizer that binauralizes only a first category audio signal based on directional information of a listener received from an audio playback device; a signal generating unit that generates the signal including the first category audio signal binauralized by the first binauralization unit and the second category audio signal that is not binauralized.

1. A voice transmission device comprising:

13. A reproducing device for reproducing a signal including audio signals of a plurality of channels, comprising: a signal acquisition unit that acquires a signal including a binauralized first category audio signal and a non-binauralized second category audio signal; a second binauralizer that binauralizes the second category audio signal acquired by the signal acquirer; a binaural generation unit that generates a binaural signal by adding together the second category sound signal binauralized by the second binaural generation unit and the first category sound signal; an audio output unit that outputs the binaural signal generated by the binaural generation unit; 16. An audio playback device comprising:

14. A sound generation and reproduction system including a sound transmission device that generates and transmits a signal including a plurality of channel sound signals, and a sound reproduction device that reproduces the signal transmitted from the sound transmission device, The audio transmission device includes: a first binauralizer that binauralizes only a first category audio signal based on directional information of a listener received from the audio playback device; a signal generating unit that generates the signal including a first category audio signal binauralized by the first binauralization unit and a second category audio signal that is not binauralized; a transmission unit that transmits the signal generated by the signal generation unit in real time, The audio reproduction device includes: a signal acquisition unit that acquires the signal generated by the voice transmission device; a second binauralizer that binauralizes the second category audio signal acquired by the signal acquirer; a binaural generation unit that generates a binaural signal by adding together the second category sound signal binauralized by the second binaural generation unit and the first category sound signal; an audio output unit that outputs the binaural signal generated by the binaural generation unit; a direction transmitting unit that transmits direction information of the listener to the audio transmitting device.

1. An audio transmission and playback system comprising:

Images