Audio signal processing method, computer program, and audio signal processing device

Abstract

Provided is an audio signal processing method executed by an audio signal processing device, the method including: an acquisition step for acquiring a first audio signal indicating a first sound, the first audio signal including first attribute information specifying an attribute of the first audio signal, and a second audio signal indicating a second sound, the second audio signal including second attribute information specifying an attribute of the second audio signal; a determination step for determining whether or not to merge the acquired first audio signal and the acquired second audio signal on the basis of an index corresponding to a first direction of arrival of a first sound at a listening position where a listener is located and a second direction of arrival of a second sound at the listening position; a merging step for generating a merged audio signal obtained by merging the first audio signal and the second audio signal when it is determined to merge the first audio signal and the second audio signal; and a playback step for outputting an output signal based on the generated merged audio signal.

Description

Audio signal processing method, computer program, and audio signal processing device

[0001] The present disclosure relates to an audio signal processing method and the like.

[0002] In recent years, products and services using ER (Extended Reality) (which may also be expressed as XR), including VR (Virtual Reality), AR (Augmented Reality), and MR (Mixed Reality), have become increasingly popular. Accordingly, audio signal processing technology that provides immersive audio to listeners in a virtual or real space by adding acoustic effects that occur according to the environment of the space to sounds emitted from a virtual sound source has become increasingly important.

[0003] The listener may also be expressed as a listener or a user. Furthermore, Patent Document 1, Patent Document 2, Patent Document 3, and Non-Patent Document 1 disclose techniques related to the audio signal processing method and the like of the present disclosure.

[0004] Japanese Patent No. 6288100 JP 2019-22049 A International Publication No. 2021 / 180938

[0005] B. C. J. Moore, "Introduction to Psychology of Hearing," Seishin Shobo, April 20, 1994, Chapter 6: Spatial Perception, p. 225. Kazuhiro Iida, "Spatial Acoustics (Acoustic Science Series)," Corona Publishing, July 2010, Figure 2.26.

[0006] However, with the technology disclosed in Patent Document 1, it may be difficult to reduce the amount of calculation and the calculation load.

[0007] Therefore, an object of the present disclosure is to provide an audio signal processing method and the like that can reduce the amount of calculation and the calculation load.

[0008] An audio signal processing method according to one aspect of the present disclosure is an audio signal processing method executed by an audio signal processing device, and includes an acquisition step of acquiring a first audio signal indicating a first sound, the first audio signal including first attribute information that specifies an attribute of the first audio signal, and a second audio signal indicating a second sound, the second audio signal including second attribute information that specifies an attribute of the second audio signal; a decision step of deciding whether to merge the acquired first audio signal and the acquired second audio signal based on an index corresponding to a first arrival direction from which the first sound arrives at a listening position where a listener is located and a second arrival direction from which the second sound arrives at the listening position; a merging step of generating a merged audio signal by merging the first audio signal and the second audio signal when it is determined to merge the first audio signal and the second audio signal; and a reproduction step of outputting an output signal based on the generated merged audio signal.

[0009] Also, an audio signal processing method according to one aspect of the present disclosure is an audio signal processing method executed by an audio signal processing device, the audio signal processing method including an acquisition step of acquiring a first audio signal indicating a first sound, the first audio signal including first attribute information specifying an attribute of the first audio signal, and a second audio signal indicating a second sound, the second audio signal including second attribute information specifying an attribute of the second audio signal; a merging step of generating a merged audio signal by merging the first audio signal and the second audio signal when it is determined to merge the acquired first audio signal and the acquired second audio signal; and and a reproduction step of outputting an output signal based on the acquired first audio signal, wherein the first attribute information included in the acquired first audio signal includes first position information indicating the position of the first sound and first volume information indicating the volume of the first sound, and the second attribute information included in the acquired second audio signal includes second position information indicating the position of the second sound and second volume information indicating the volume of the second sound, and the merging step determines a merge sound arrival direction in which the merge sound indicated by the merge audio signal arrives at a listening position where a listener is located, based on the first position information and the first volume information and the second position information and the second volume information.

[0010] In addition, an audio signal processing method according to one aspect of the present disclosure is an audio signal processing method executed by an audio signal processing device, and includes an acquisition step of acquiring M (M is an integer greater than or equal to 2) audio signals indicating predetermined sounds, each of which includes attribute information that identifies the attributes of the audio signals; a decision step of deciding whether to merge N (N is an integer greater than or equal to 1 and less than or equal to M) audio signals out of the acquired M audio signals based on the direction of arrival of each of the M predetermined sounds at a listening position where a listener is located; a merging step of generating a merged audio signal by merging the N audio signals when it is decided to merge the N audio signals; and a playback step of outputting an output signal based on the generated merged audio signal.

[0011] Further, an audio signal processing method according to an aspect of the present disclosure is an audio signal processing method executed by an audio signal processing device, and includes: an acquisition step of acquiring a first audio signal indicating a first sound, the first audio signal including first attribute information specifying an attribute of the first audio signal; and a second audio signal indicating a second sound, the second audio signal including second attribute information specifying an attribute of the second audio signal; a decision step of deciding whether to merge the acquired first audio signal and the acquired second audio signal based on a first arrival direction from which the first sound arrives at a listening position where a listener is located and a second arrival direction from which the second sound arrives at the listening position; a merging step of generating a merged audio signal by merging the first audio signal and the second audio signal when it is decided to merge the first audio signal and the second audio signal; and a reproduction step of outputting an output signal based on the generated merged audio signal, wherein in the decision step, Among the plurality of pieces of arrival direction information, one piece of arrival direction information corresponding to the first arrival direction and one piece of arrival direction information corresponding to the second arrival direction are determined, and in the merging step, when the piece of arrival direction information corresponding to the determined first arrival direction and the piece of arrival direction information corresponding to the determined second arrival direction are the same, the merged audio signal is generated, and in the reproducing step, the output signal generated by applying the head-related transfer function indicated by the piece of arrival direction information corresponding to the first arrival direction to the generated merged audio signal is output, and when the piece of arrival direction information corresponding to the determined first arrival direction and the piece of arrival direction information corresponding to the determined second arrival direction are different, the output signal generated by applying the head-related transfer function indicated by the piece of arrival direction information corresponding to the first arrival direction to the first audio signal and the output signal generated by applying the head-related transfer function indicated by the piece of arrival direction information corresponding to the second arrival direction to the second audio signal are output.

[0012] Furthermore, a computer program according to one aspect of the present disclosure causes a computer to execute the above-described audio signal processing method.

[0013] Furthermore, an audio signal processing device according to one aspect of the present disclosure includes an acquisition unit that acquires a first audio signal indicating a first sound, the first audio signal including first attribute information that identifies an attribute of the first audio signal, and a second audio signal indicating a second sound, the second audio signal including second attribute information that identifies an attribute of the second audio signal; a decision unit that decides whether to merge the acquired first audio signal and the acquired second audio signal based on an index corresponding to a first arrival direction from which the first sound arrives at a listening position where a listener is located and a second arrival direction from which the second sound arrives at the listening position; a merge unit that generates a merged audio signal by merging the first audio signal and the second audio signal when it is determined to merge the first audio signal and the second audio signal; and a playback unit that outputs an output signal based on the generated merged audio signal.

[0014] These comprehensive or specific aspects may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0015] According to an audio signal processing method and the like according to one aspect of the present disclosure, the amount of calculation and the calculation load can be reduced.

[0016] FIG. 1 is a diagram showing an example of direct sound and reflected sound generated in a sound space. FIG. 2 is a diagram showing an example of a stereophonic sound reproduction system according to Embodiment 1. FIG. 3A is a block diagram showing an example of the configuration of an encoding device according to Embodiment 1. FIG. 3B is a block diagram showing an example of the configuration of a decoding device according to Embodiment 1. FIG. 3C is a block diagram showing another example of the configuration of an encoding device according to Embodiment 1. FIG. 3D is a block diagram showing another example of the configuration of a decoding device according to Embodiment 1. FIG. 4A is a block diagram showing an example of the configuration of a decoder according to Embodiment 1. FIG. 4B is a block diagram showing another example of the configuration of a decoder according to Embodiment 1. FIG. 5 is a diagram showing an example of the physical configuration of an audio signal processing device according to Embodiment 1. FIG. 6 is a diagram showing an example of the physical configuration of an encoding device according to Embodiment 1. FIG. 7 is a block diagram showing an example of the configuration of a rendering unit according to Embodiment 1. FIG. 8 is a flowchart showing an example of the operation of the audio signal processing device according to Embodiment 1. FIG. 9 is a diagram showing a positional relationship between a listener and an obstacle object when they are relatively far apart. FIG. 10 is a diagram showing a positional relationship between a listener and an obstacle object when they are relatively close together. FIG. 11 is a diagram showing the relationship between the time difference between direct sound and reflected sound and a threshold. FIG. 12A is a diagram showing a part of an example of a method for setting threshold data. FIG. 12B is a diagram showing part of an example of a method for setting threshold data. FIG. 12C is a diagram showing part of an example of a method for setting threshold data. FIG. 13 is a diagram showing an example of a method for setting thresholds. FIG. 14 is a flowchart showing an example of a selection process. FIG. 15 is a diagram showing the relationship between the direction of a direct sound, the direction of a reflected sound, the time difference, and the threshold. FIG. 16 is a diagram showing the relationship between the angle difference, the time difference, and the threshold. FIG. 17 is a block diagram showing another example of the configuration of a rendering unit. FIG. 18 is a flowchart showing another example of the selection process. FIG. 19 is a flowchart showing yet another example of the selection process. FIG. 20 is a flowchart showing a first modified example of the operation of the audio signal processing device according to Embodiment 1. FIG. 21 is a flowchart showing a second modified example of the operation of the audio signal processing device according to Embodiment 1. FIG. 22 is a diagram showing an example of the arrangement of an avatar, a sound source object, and an obstacle object. FIG. 23 is a flowchart showing yet another example of the selection process.FIG. 24 is a block diagram showing an example of a configuration for a rendering unit to perform pipeline processing. FIG. 25 is a diagram showing transmission and diffraction of sound. FIG. 26 is a diagram showing an example of a positional relationship between a listener and an obstacle object according to Embodiment 1. FIG. 27 is a diagram showing another example of a positional relationship between a listener and an obstacle object according to Embodiment 1. FIG. 28 is an example of an echo detection limit threshold according to Embodiment 1. FIG. 29 is a diagram showing an example in which the sound image position of a reflected sound is moved in the positional relationship shown in FIG. 27. FIG. 30 is a diagram showing an example in which direct sound and indirect sound arrive at a listener from the same direction. FIG. 31 is a diagram showing an example in which direct sound and indirect sound arrive at a listener from one sound source and another sound source, respectively. FIG. 32 is a diagram showing an example in which transmitted sound and diffracted sound, which are sounds emitted from one sound source, arrive at a listener. FIG. 33 is a block diagram showing an example of a configuration of a rendering unit according to Embodiment 2. FIG. 34 is a flowchart showing Operation Example 1 of the audio signal processing device according to Embodiment 2. FIG. 35 is a flowchart showing an operation example of processing performed by the selection unit and the reproduction unit in Operation Example 1 according to Embodiment 2. 36 is a flowchart showing an example of processing performed by a selection unit and a playback unit in operation example 2 according to embodiment 2. FIG. 37 is a diagram showing an SOFA of an HRTF according to embodiment 2. FIG. 38 is a diagram showing a cone for explaining the relationship between the position at which an HRTF is defined and the listening position according to embodiment 2. FIG. 39 is a flowchart showing an example of processing performed by a selection unit and a playback unit in operation example 3 according to embodiment 2. FIG. 40 is a diagram for explaining the arrival direction of a merge sound used in operation example 3 according to embodiment 2. FIG. 41 is a flowchart showing an example of processing performed by a selection unit and a playback unit in operation example 4 according to embodiment 2. FIG. 42 is a flowchart showing an example of processing performed by a selection unit and a playback unit in operation example 5 according to embodiment 2. FIG. 43 is a flowchart showing an example of processing performed by a selection unit and a playback unit in operation example 6 according to embodiment 2. FIG. 44 is a flowchart showing an example of processing performed by a selection unit and a playback unit in operation example 7 according to embodiment 2. FIG. 45 is a block diagram showing an example of the configuration of a rendering unit according to embodiment 3.FIG. 46 is a diagram showing an example of a merge region according to Embodiment 3. FIG. 47 is a flowchart showing operation example 1 of the audio signal processing device according to Embodiment 3. FIG. 48 is a flowchart showing an operation example of processing performed by the selection unit and the reproduction unit in operation example 1 according to Embodiment 3. FIG. 49 is a diagram for explaining a merge region according to Embodiment 3. FIG. 50 is another diagram for explaining a merge region according to Embodiment 3. FIG. 51 is a diagram for explaining a method for calculating a merge sound arrival direction according to Embodiment 3. FIG. 52 is a diagram showing another first example of a merge region according to Embodiment 3. FIG. 53 is a diagram showing another second example of a merge region according to Embodiment 3. FIG. 54 is a diagram showing another third example of a merge region according to Embodiment 3. FIG. 55 is a diagram showing another fourth example of a merge region according to Embodiment 3. FIG. 56 is a diagram showing another fifth example of a merge region according to Embodiment 3. FIG. 57 is a diagram showing another sixth example of a merge region according to Embodiment 3. FIG. 58 is a diagram showing an example for explaining a phase difference between a direct sound and a reflected sound according to Embodiment 4. FIG. 59 is a diagram showing another example for explaining a phase difference between a direct sound and a reflected sound according to Embodiment 4. Fig. 60 is a flowchart showing an example of operation of a process performed by a selection unit in operation example 1 according to embodiment 4. Fig. 61 is a flowchart showing a first example of details of the process of step S703 according to embodiment 4. Fig. 62 is a diagram showing direct sound, reflected sound, and a sound obtained by combining direct sound and reflected sound according to embodiment 4. Fig. 63 is a flowchart showing a third example of details of the process of step S703 according to embodiment 4. Fig. 64 is a diagram showing a first example of an HRTF SOFA according to another embodiment. Fig. 65 is a diagram showing a second example of an HRTF SOFA according to another embodiment. Fig. 66 is a diagram showing a third example of an HRTF SOFA according to another embodiment.

[0017] (Knowledge forming the basis of the present disclosure) Conventionally, audio signal processing techniques have been studied that provide immersive audio to listeners in a virtual or real space by adding acoustic effects that arise according to the environment of the space to sounds emitted by a virtual sound source.

[0018] Such an audio signal processing technology is disclosed in Patent Document 1. More specifically, Patent Document 1 discloses a technology for detecting the importance of an audio signal (voice signal) and not outputting an audio signal with a detected low importance. By not outputting an audio signal with a low importance in this way, the audio signal processing technology is expected to reduce the amount of calculation and the calculation load.

[0019] Incidentally, in a sound space (virtual space or real space), reflected sound can be important.

[0020] 1 is a diagram showing an example of direct sound and reflected sound generated in a sound space. In acoustic processing that expresses the characteristics of a virtual space with sound, it is effective to reproduce not only direct sound but also reflected sound in order to express the size of the space, the material of the walls, etc., and to accurately grasp the position of the sound source (localization of the sound image).

[0021] For example, when listening to sound in a rectangular room as shown in Figure 1, six primary reflections are generated for a single sound source, corresponding to the six walls. Reproducing these reflections provides clues for a proper understanding of the space and sound image. Furthermore, for each reflection, secondary reflections are generated from surfaces other than the surface that generated the reflection. These reflections also provide useful perceptual clues.

[0022] However, even if only secondary reflections are taken into account, one sound source will produce one direct sound and 36 (6 + 6 x 5) reflected sounds, resulting in 37 sound rays, and a considerable amount of calculation is required to process these sound rays.

[0023] Furthermore, in recent applications envisioned for the Metaverse, such as virtual meetings, virtual shopping, or virtual concerts, multiple sound sources will inevitably be present, requiring even greater amounts of computation.

[0024] In addition, listeners who listen to sounds in a virtual space use headphones or VR goggles. To provide such listeners with stereophonic sound, binaural processing is performed on each sound ray, which provides a sound pressure ratio and phase difference between the two ears to reproduce the direction of sound arrival and the sense of perspective. Therefore, if all reflected sounds are to be reproduced, the amount of calculation required becomes enormous.

[0025] On the other hand, for convenience, small storage batteries are sometimes used as the batteries for VR goggles worn by listeners who experience virtual space. In order to extend the battery life, it is desirable to reduce the computational load required for the above-mentioned processing. To achieve this, it is desirable to reduce the number of sound rays, which may number on the order of several hundred, to a degree that does not impair sound localization and spatial understanding.

[0026] Furthermore, in some sound reproduction systems, degrees of freedom such as 6 DoF (6 Degrees of Freedom) are permitted for the position and orientation of the listener (i.e., the listening position where the listener is located). In this case, the positional relationship between the listener, the sound source, and the object that reflects the sound is not determined until playback (rendering). Therefore, reflected sounds are also not determined until playback. Therefore, it is difficult to determine the reflected sounds to be processed in advance.

[0027] Therefore, selecting and outputting (reproducing) one or more reflected sounds that are to be processed or not to be processed from among the multiple reflected sounds that occur in the sound space during playback is useful for appropriately reducing the amount of calculation and the calculation load.

[0028] Note that controlling whether to select a sound corresponds to determining whether to select a sound, and more specifically, to determining whether to select a sound and output (play) it. Furthermore, selecting a sound may mean selecting the sound as a sound to be processed, or may mean selecting the sound as a sound not to be processed.

[0029] However, in Patent Document 1, the importance of an audio signal, more specifically, the importance of a direct sound indicated by the audio signal is detected, but the importance of a reflected sound is not considered. Therefore, when an indirect sound such as a reflected sound occurs as shown in Figure 1, the amount of calculation and the calculation load increase, which means that it may be difficult to reduce the amount of calculation and the calculation load.

[0030] Therefore, there is a demand for an audio signal processing method that can reduce the amount of calculation and the calculation load in the sound space.

[0031] Therefore, an audio signal processing method according to a first aspect of the present disclosure is an audio signal processing method executed by an audio signal processing device, and includes: an acquisition step of acquiring a first audio signal indicating a first sound, the first audio signal including first attribute information that specifies an attribute of the first audio signal; and a second audio signal indicating a second sound, the second audio signal including second attribute information that specifies an attribute of the second audio signal; a decision step of deciding whether to merge the acquired first audio signal and the acquired second audio signal based on an index corresponding to a first arrival direction from which the first sound arrives at a listening position where a listener is located and a second arrival direction from which the second sound arrives at the listening position; a merging step of generating a merged audio signal by merging the first audio signal and the second audio signal when it is decided to merge the first audio signal and the second audio signal; and a reproduction step of outputting an output signal based on the generated merged audio signal.

[0032] As a result, it is determined whether the first audio signal and the second audio signal are merged based on an index corresponding to the first arrival direction of the first sound and the second arrival direction of the second sound. If it is determined that the first audio signal and the second audio signal are merged, the reproducing step outputs an output signal based on the merged audio signal. If it is determined that the first audio signal and the second audio signal are not merged, the reproducing step outputs an output signal based on the first audio signal and an output signal based on the second audio signal. Compared to when both the output signal based on the first audio signal and the output signal based on the second audio signal are output, when the output signal based on the merged audio signal is output, the number of output signals to be output is reduced, thereby reducing the amount of calculation and the calculation load. In other words, it is possible to realize an audio signal processing method that can reduce the amount of calculation and the calculation load.

[0033] An audio signal processing method according to a second aspect of the present disclosure is the audio signal processing method according to the first aspect, wherein the index is an index consisting of two orthogonal axes.

[0034] This allows the use of simple indices, i.e., does not require a large amount of calculation and a large calculation load compared to when complex indices are used, thereby realizing an audio signal processing method that can reduce the amount of calculation and the calculation load.

[0035] A third aspect of the present disclosure provides an audio signal processing method in which, in the audio signal processing method according to the first or second aspect, the first attribute information included in the acquired first audio signal includes first position information indicating the position of the first sound and first volume information indicating the volume of the first sound, and the second attribute information included in the acquired second audio signal includes second position information indicating the position of the second sound and second volume information indicating the volume of the second sound, and in the merging step, a direction of arrival of the merge sound indicated by the merge audio signal at the listening position is determined based on the first position information and the first volume information and the second position information and the second volume information.

[0036] As a result, in the merging step, the direction from which the merging sound comes can be determined based on the first position information and the first volume information and the second position information and the second volume information.

[0037] An audio signal processing method according to a fourth aspect of the present disclosure is the audio signal processing method according to the third aspect, wherein in the merging step, when the volume of the first sound indicated by the first volume information is regarded as a weight corresponding to the position of the first sound indicated by the first position information and the volume of the second sound indicated by the second volume information is regarded as a weight corresponding to the position of the second sound indicated by the second position information, the position of the merged sound is determined to be the position of the center of gravity of the first sound and the second sound, and the direction from the determined position of the center of gravity toward the listening position is determined to be the direction from which the merged sound arrives.

[0038] This allows the merge sound arrival direction to be determined based on the positions of the centers of gravity of the first sound and the second sound, which means that the merge sound arrival direction can be determined easily. In other words, since determining the merge sound arrival direction does not require a large amount of calculation and a large calculation load, it is possible to realize an audio signal processing method that can reduce the amount of calculation and the calculation load.

[0039] A sound signal processing method according to a fifth aspect of the present disclosure is a sound signal processing method executed by a sound signal processing device, the sound signal processing method including: an acquisition step of acquiring a first sound signal indicating a first sound, the first sound signal including first attribute information specifying an attribute of the first sound signal; and a second sound signal indicating a second sound, the second sound signal including second attribute information specifying an attribute of the second sound signal; a merging step of generating a merged sound signal by merging the first sound signal and the second sound signal when it is determined to merge the acquired first sound signal and the acquired second sound signal; and a merging step of generating a merged sound signal by merging the first sound signal and the second sound signal. and a reproduction step of outputting an output signal based on the acquired first audio signal, wherein the first attribute information included in the acquired first audio signal includes first position information indicating the position of the first sound and first volume information indicating the volume of the first sound, and the second attribute information included in the acquired second audio signal includes second position information indicating the position of the second sound and second volume information indicating the volume of the second sound, and in the merging step, a merge sound arrival direction in which the merge sound indicated by the merge audio signal arrives at a listening position where a listener is located is determined based on the first position information and the first volume information and the second position information and the second volume information.

[0040] As a result, if it is determined that the first audio signal and the second audio signal are merged, the reproducing step outputs an output signal based on the merged audio signal. If it is determined that the first audio signal and the second audio signal are not merged, the reproducing step outputs an output signal based on the first audio signal and an output signal based on the second audio signal. Compared to when both the output signal based on the first audio signal and the output signal based on the second audio signal are output, when the output signal based on the merged audio signal is output, the number of output signals to be output is reduced, thereby reducing the amount of calculation and the calculation load. In other words, an audio signal processing method that can reduce the amount of calculation and the calculation load can be realized.

[0041] An audio signal processing method according to a sixth aspect of the present disclosure is an audio signal processing method according to any one of the second to fifth aspects, in which, when it is decided to merge the first audio signal and the second audio signal, in the playback step, the output signal generated by applying a head-related transfer function based on the determined direction from which the merge sound arrives to the generated merged audio signal is output.

[0042] This makes it possible to realize an audio signal processing method that allows the listener to hear merged sounds with a more realistic feel.

[0043] In an audio signal processing method according to a seventh aspect of the present disclosure, when it is determined not to merge the first audio signal and the second audio signal in the audio signal processing method according to any one of the first to fourth aspects, the reproduction step outputs the output signal generated by applying a head-related transfer function based on the first direction of arrival to the first audio signal, and the output signal generated by applying a head-related transfer function based on the second direction of arrival to the second audio signal.

[0044] This makes it possible to realize an audio signal processing method that allows the listener to hear the first sound and the second sound with a more realistic feel.

[0045] An audio signal processing method according to an eighth aspect of the present disclosure is an audio signal processing method executed by an audio signal processing device, and includes an acquisition step of acquiring M (M is an integer greater than or equal to 2) audio signals indicating predetermined sounds and including attribute information that identifies the attributes of the audio signals; a decision step of deciding whether to merge N (N is an integer greater than or equal to 1 and less than or equal to M) audio signals out of the M acquired audio signals based on the direction of arrival of each of the M predetermined sounds at a listening position where a listener is located; a merging step of generating a merged audio signal by merging the N audio signals when it is decided to merge the N audio signals; and a playback step of outputting an output signal based on the generated merged audio signal.

[0046] As a result, it is determined whether or not the N audio signals are merged based on the arrival direction of the predetermined sound. If it is determined that the N audio signals are merged, the reproduction step outputs an output signal based on the merged audio signal. If it is determined that the N audio signals are not merged, the reproduction step outputs output signals based on each of the N audio signals. Compared to when output signals based on each of the N audio signals are output, when an output signal based on the merged audio signal is output, the number of output signals is reduced, thereby reducing the amount of calculation and the calculation load. In other words, it is possible to realize an audio signal processing method that can reduce the amount of calculation and the calculation load.

[0047] A ninth aspect of the present disclosure is an audio signal processing method according to the eighth aspect, wherein the attribute information contained in each of the M acquired audio signals includes predetermined sound position information indicating the position of the predetermined sound and predetermined sound volume information indicating the volume of the predetermined sound, and in the merging step, the direction of arrival of the merge sound indicated by the merge audio signal at the listening position is determined based on the N pieces of predetermined sound position information and the N pieces of predetermined sound volume information.

[0048] As a result, in the merging step, the direction from which the merging sound comes can be determined based on the predetermined sound position information and the predetermined sound volume information.

[0049] An audio signal processing method according to a tenth aspect of the present disclosure is the audio signal processing method according to the ninth aspect, wherein in the merging step, when the volume of the specified sound indicated by the specified sound volume information is regarded as a weight corresponding to the position of the specified sound indicated by the specified sound position information for the specified sound, the position of the merging sound is determined to be the position of the center of gravity of the N specified sounds, and the direction from the determined position of the center of gravity toward the listening position is determined to be the direction from which the merging sound arrives.

[0050] This allows the merge sound arrival direction to be determined based on the position of the center of gravity of the N predetermined sounds, which means that the merge sound arrival direction can be determined easily. In other words, since determining the merge sound arrival direction does not require a large amount of calculation and a large calculation load, it is possible to realize an audio signal processing method that can reduce the amount of calculation and the calculation load.

[0051] An audio signal processing method according to an eleventh aspect of the present disclosure is an audio signal processing method executed by an audio signal processing device, the audio signal processing method including: an acquisition step of acquiring a first audio signal indicating a first sound, the first audio signal including first attribute information specifying an attribute of the first audio signal; and a second audio signal indicating a second sound, the second audio signal including second attribute information specifying an attribute of the second audio signal; a decision step of deciding whether to merge the acquired first audio signal and the acquired second audio signal based on a first arrival direction from which the first sound arrives at a listening position where a listener is located and a second arrival direction from which the second sound arrives at the listening position; a merging step of generating a merged audio signal by merging the first audio signal and the second audio signal when it is decided to merge the first audio signal and the second audio signal; and a reproduction step of outputting an output signal based on the generated merged audio signal, wherein in the decision step, Among a plurality of pieces of arrival direction information, one piece of arrival direction information corresponding to the first arrival direction and one piece of arrival direction information corresponding to the second arrival direction are determined. In the merging step, if the piece of arrival direction information corresponding to the determined first arrival direction and the piece of arrival direction information corresponding to the determined second arrival direction are the same, the merged audio signal is generated. In the reproducing step, the output signal generated by applying the head-related transfer function indicated by the piece of arrival direction information corresponding to the first arrival direction to the generated merged audio signal is output. In the case where the piece of arrival direction information corresponding to the determined first arrival direction and the piece of arrival direction information corresponding to the determined second arrival direction are different, the output signal generated by applying the head-related transfer function indicated by the piece of arrival direction information corresponding to the first arrival direction to the first audio signal and the output signal generated by applying the head-related transfer function indicated by the piece of arrival direction information corresponding to the second arrival direction to the second audio signal are output.

[0052] As a result, it is determined whether the first audio signal and the second audio signal are merged based on arrival direction information corresponding to the first arrival direction of the first sound and arrival direction information corresponding to the second arrival direction of the second sound. If it is determined that the first audio signal and the second audio signal are merged, the reproducing step outputs an output signal based on the merged audio signal. If it is determined that the first audio signal and the second audio signal are not merged, the reproducing step outputs an output signal based on the first audio signal and an output signal based on the second audio signal. Compared to when both the output signal based on the first audio signal and the output signal based on the second audio signal are output, when the output signal based on the merged audio signal is output, the number of output signals to be output is reduced, thereby reducing the amount of calculation and the calculation load. In other words, it is possible to realize an audio signal processing method that can reduce the amount of calculation and the calculation load.

[0053] A computer program according to a twelfth aspect of the present disclosure is a computer program for causing a computer to execute the audio signal processing method according to any one of the first to eleventh aspects.

[0054] This allows the computer to execute the above-described audio signal processing method in accordance with the computer program.

[0055] An audio signal processing device according to a twelfth aspect of the present disclosure includes an acquisition unit that acquires a first audio signal indicating a first sound, the first audio signal including first attribute information that specifies an attribute of the first audio signal, and a second audio signal indicating a second sound, the second audio signal including second attribute information that specifies an attribute of the second audio signal; a decision unit that decides whether to merge the acquired first audio signal and the acquired second audio signal based on an index corresponding to a first arrival direction from which the first sound arrives at a listening position where a listener is located and a second arrival direction from which the second sound arrives at the listening position; a merge unit that, when it is decided to merge the first audio signal and the second audio signal, generates a merged audio signal by merging the first audio signal and the second audio signal; and a playback unit that outputs an output signal based on the generated merged audio signal.

[0056] As a result, it is determined whether the first audio signal and the second audio signal are merged based on an index corresponding to the first arrival direction of the first sound and the second arrival direction of the second sound. If it is determined that the first audio signal and the second audio signal are merged, the reproduction unit outputs an output signal based on the merged audio signal. If it is determined that the first audio signal and the second audio signal are not merged, the reproduction unit outputs an output signal based on the first audio signal and an output signal based on the second audio signal. Compared to when both the output signal based on the first audio signal and the output signal based on the second audio signal are output, when the output signal based on the merged audio signal is output, the number of output signals to be output is reduced, thereby reducing the amount of calculation and the calculation load. In other words, it is possible to realize an audio signal processing device that can reduce the amount of calculation and the calculation load.

[0057] (Embodiment 1) (Example of a stereophonic sound reproduction system) Fig. 2 is a diagram showing an example of a stereophonic sound reproduction system 1000. Specifically, Fig. 2 shows the stereophonic sound reproduction system 1000, which is an example of a system to which the acoustic processing or decoding processing of the present disclosure can be applied. Stereophonic sound is also expressed as immersive audio. The stereophonic sound reproduction system 1000 includes an audio signal processing device 1001 and an audio presentation device 1002.

[0058] The audio signal processing device 1001, also referred to as an audio processing device, performs audio processing on an audio signal emitted by a virtual sound source to generate an audio signal after the audio processing to be presented to a listener. The audio signal is not limited to a voice, and may be any audible sound. The audio processing is, for example, signal processing performed on the audio signal to reproduce one or more effects that the sound undergoes from the time it is generated by the sound source until it reaches the listener.

[0059] The audio signal processing device 1001 performs acoustic processing based on spatial information that describes factors that cause the above-mentioned effects. The spatial information includes, for example, information indicating the positions of a sound source, a listener, and surrounding objects, information indicating the shape of a space, and parameters related to sound propagation. The audio signal processing device 1001 is, for example, a PC (Personal Computer), a smartphone, a tablet, a game console, or the like.

[0060] The signal after acoustic processing is presented to the listener from the audio presentation device 1002. The audio presentation device 1002 is connected to the audio signal processing device 1001 via wireless or wired communication. The audio signal after acoustic processing generated by the audio signal processing device 1001 is transmitted to the audio presentation device 1002 via wireless or wired communication.

[0061] When the audio presentation device 1002 is configured with a plurality of devices, such as a device for the right ear and a device for the left ear, the plurality of devices present sounds in synchronization through communication between the plurality of devices or communication between each of the plurality of devices and the audio signal processing device 1001. The audio presentation device 1002 is, for example, headphones, earphones, or a head-mounted display worn on the head of a listener, or a surround speaker configured with a plurality of fixed speakers.

[0062] The stereophonic sound reproduction system 1000 may be used in combination with an image presentation device or a stereoscopic video presentation device that provides a visual ER experience, including AR / VR. For example, the space handled by the spatial information is a virtual space, and the positions of a sound source, a listener, and an object in the space are the virtual positions of a virtual sound source, a virtual listener, and a virtual object in the virtual space. The space may also be expressed as a sound space. The spatial information may also be expressed as sound space information.

[0063] 2 shows an example of a system configuration in which the audio signal processing device 1001 and the audio presentation device 1002 are separate devices, but the stereophonic sound reproduction system 1000 to which the audio processing method (audio signal processing method) or decoding method of the present disclosure can be applied is not limited to the configuration shown in Fig. 2. For example, the audio signal processing device 1001 may be included in the audio presentation device 1002, which may perform both audio processing and sound presentation.

[0064] The acoustic processing described in the present disclosure may be shared between the audio signal processing device 1001 and the audio presentation device 1002. A server connected to the audio signal processing device 1001 or the audio presentation device 1002 via a network may perform part or all of the acoustic processing described in the present disclosure.

[0065] Furthermore, the audio signal processing device 1001 may perform audio processing by decoding a bit stream generated by encoding at least a portion of the data of the audio signal and spatial information used for the audio processing. Therefore, the audio signal processing device 1001 may be referred to as a decoding device.

[0066] (Example of Encoding Device) Fig. 3A is a block diagram showing an example configuration of encoding device 1100. Specifically, Fig. 3A shows the configuration of encoding device 1100, which is an example of an encoding device of the present disclosure.

[0067] Input data 1101 is data to be coded, including spatial information and / or an audio signal, that is input to an encoder 1102. Details of the spatial information will be explained later.

[0068] The encoder 1102 encodes the input data 1101 to generate encoded data 1103. The encoded data 1103 is, for example, a bit stream generated by the encoding process.

[0069] The memory 1104 stores the encoded data 1103. The memory 1104 may be, for example, a hard disk or a solid-state drive (SSD), or may be other memory.

[0070] In the above description, a bitstream generated by an encoding process is given as an example of the encoded data 1103 stored in memory 1104, but the encoded data 1103 may be data other than a bitstream. For example, the encoding device 1100 may store converted data generated by converting a bitstream into a predetermined data format in memory 1104. The converted data may be, for example, a file or a multiplexed stream corresponding to one or more bitstreams.

[0071] Here, the file is a file having a file format such as ISO Base Media File Format (ISOBMFF), etc. The encoded data 1103 may be in the form of a plurality of packets generated by dividing the bit stream or file.

[0072] For example, the bitstream generated by the encoder 1102 may be converted into data different from the bitstream. In this case, the encoding device 1100 may include a conversion unit (not shown) and perform the conversion process in the conversion unit, or may perform the conversion process in a CPU (Central Processing Unit), which is an example of a processor described later.

[0073] (Example of Decoding Device) Fig. 3B is a block diagram showing an example configuration of the decoding device 1110. Specifically, Fig. 3B shows the configuration of the decoding device 1110, which is an example of a decoding device of the present disclosure.

[0074] The memory 1114 stores, for example, the same data as the coded data 1103 generated by the coding device 1100. The stored data is read from the memory 1114 and input to the decoder 1112 as input data 1113. The input data 1113 is, for example, a bitstream to be decoded. The memory 1114 may be, for example, a hard disk or an SSD, or may be some other memory.

[0075] Note that the decoding device 1110 may convert the data read from the memory 1114 and input the converted data to the decoder 1112 as input data 1113, rather than inputting the data directly to the decoder 1112 as input data 1113. The data before conversion may be, for example, multiplexed data including one or more bitstreams. Here, the multiplexed data may be a file having a file format such as ISOBMFF.

[0076] The data before conversion may also be a plurality of packets generated by dividing the bitstream or file. Data different from the bitstream may be read from memory 1114 and converted into a bitstream. In this case, decoding device 1110 may include a conversion unit (not shown) and perform the conversion process, or a CPU (an example of a processor, described later) may perform the conversion process.

[0077] Decoder 1112 decodes input data 1113 to produce an audio signal 1111 representing the audio to be presented to the listener.

[0078] (Another Example of Encoding Device) Fig. 3C is a block diagram showing another example of the configuration of an encoding device. Specifically, Fig. 3C shows the configuration of encoding device 1120, which is another example of an encoding device of the present disclosure. In Fig. 3C, the same components as those in Fig. 3A are assigned the same reference numerals as those in Fig. 3A, and descriptions of these components will be omitted.

[0079] Coding device 1100 stores coded data 1103 in memory 1104. On the other hand, coding device 1120 differs from coding device 1100 in that coding device 1120 includes a transmitting unit 1121 that transmits coded data 1103 to the outside.

[0080] The transmitter 1121 transmits to another device or a server a transmission signal 1122 generated based on the encoded data 1103 or data converted into another data format from the encoded data 1103. The data used to generate the transmission signal 1122 is, for example, the bit stream, multiplexed data, file, or packet described in the encoding device 1100.

[0081] (Another Example of Decoding Device) Fig. 3D is a block diagram showing another example of the configuration of a decoding device. Specifically, Fig. 3D shows the configuration of a decoding device 1130, which is another example of a decoding device of the present disclosure. In Fig. 3D, the same components as those in Fig. 3B are assigned the same reference numerals as those in Fig. 3B, and descriptions of these components will be omitted.

[0082] The decoding device 1110 reads input data 1113 from a memory 1114. On the other hand, the decoding device 1130 differs from the decoding device 1110 in that it includes a receiving unit 1131 that receives the input data 1113 from an external source.

[0083] The receiving unit 1131 receives a received signal 1132, acquires received data, and outputs input data 1113 to be input to the decoder 1112. The received data may be the same as the input data 1113 to be input to the decoder 1112, or may be data in a data format different from that of the input data 1113.

[0084] If the data format of the received data is different from the data format of the input data 1113, the receiving unit 1131 may convert the received data into the input data 1113. Alternatively, a conversion unit or a CPU (not shown) of the decoding device 1130 may convert the received data into the input data 1113. The received data is, for example, a bit stream, multiplexed data, a file, or a packet, as described in the encoding device 1120.

[0085] (Example of Decoder) Fig. 4A is a block diagram showing an example of the configuration of the decoder 1200. Specifically, Fig. 4A shows the configuration of the decoder 1200, which is an example of the decoder 1112 in Fig. 3B or 3D.

[0086] The input data 1113 is an encoded bitstream, and includes encoded audio data, which is an encoded audio signal, and metadata used in acoustic processing.

[0087] The spatial information management unit 1201 acquires and analyzes metadata included in the input data 1113. The metadata includes information describing elements that act on sounds arranged in a sound space. The spatial information management unit 1201 manages spatial information used for acoustic processing obtained by analyzing the metadata, and provides the spatial information to the rendering unit 1203.

[0088] In the present disclosure, the information used for acoustic processing is expressed as spatial information, but other expressions may be used. For example, the information used for acoustic processing may be expressed as sound space information or scene information. Furthermore, when the information used for acoustic processing changes over time, the spatial information input to the rendering unit 1203 may be information expressed as a spatial state, a sound space state, a scene state, or the like.

[0089] Furthermore, the spatial information may be managed for each sound space or each scene. For example, when a plurality of different rooms are represented as virtual spaces, the rooms may be managed as a plurality of different scenes. Furthermore, the spatial information may be managed as different scenes depending on the situation represented in the same space.

[0090] Therefore, a plurality of pieces of spatial information may be managed for a plurality of sound spaces or a plurality of scenes. In managing the plurality of pieces of spatial information, an identifier for identifying each piece of spatial information may be assigned to the spatial information.

[0091] The spatial information data may be included in a bitstream, which is an example of input data 1113. Alternatively, the bitstream may include an identifier of the spatial information, and the spatial information data may be acquired from an information source other than the bitstream. Specifically, when the bitstream includes only the identifier of the spatial information, the identifier of the spatial information may be used in rendering to acquire the spatial information data stored in a memory within the device or an external server as input data 1113.

[0092] It should be noted that the information managed by the spatial information management unit 1201 is not limited to information included in the bitstream. For example, the input data 1113 may include data that is not included in the bitstream and indicates the characteristics and structure of a space acquired from software or a server that provides VR or AR.

[0093] The input data 1113 may also include data indicating the characteristics and positions of listeners or objects, etc. The input data 1113 may also include information about the positions of listeners acquired by sensors provided in the terminal including the decoding device (1110, 1130), or may include information indicating the position of the terminal estimated based on the information acquired by the sensors.

[0094] That is, the spatial information management unit 1201 may communicate with an external system or server to acquire spatial information and listener positions (i.e., listening positions). The spatial information management unit 1201 may also acquire clock synchronization information from the external system and execute processing to synchronize with the clock of the rendering unit 1203.

[0095] Note that the space in the above description may be a virtually formed space, i.e., a VR space, or may be a real space or a virtual space corresponding to a real space, i.e., an AR space or an MR space. The virtual space may also be expressed as a sound field or a sound space. Furthermore, the information indicating a position in the above description may be information such as coordinate values ​​indicating a position within a space, information indicating a relative position with respect to a predetermined reference position, or information indicating the movement or acceleration of a position within a space.

[0096] The audio data decoder 1202 decodes the encoded audio data included in the input data 1113 to obtain an audio signal.

[0097] The encoded audio data acquired by the stereophonic sound reproduction system 1000 is a bitstream encoded in a predetermined format such as MPEG-H 3D Audio (ISO / IEC 23008-3). Note that MPEG-H 3D Audio is merely one example of an encoding method that can be used to generate the encoded audio data contained in the bitstream. The encoded audio data may also be a bitstream encoded using another encoding method.

[0098] For example, the encoding method may be a lossy codec such as MP3 (MPEG-1 Audio Layer-3), AAC (Advanced Audio Coding), WMA (Windows Media Audio), AC3 (Audio Codec-3), or Vorbis. Alternatively, the encoding method may be a lossless codec such as ALAC (Apple Lossless Audio Codec) or FLAC (Free Lossless Audio Codec).

[0099] Alternatively, any other encoding method may be used. For example, PCM data may be a type of encoded audio data. In this case, the decoding process may be, for example, a process of converting an N-bit binary number into a number format (e.g., floating-point format) that can be processed by the rendering unit 1203, where the number of quantization bits of the PCM data is N.

[0100] The rendering unit 1203 acquires the audio signal and spatial information, performs acoustic processing on the audio signal using the spatial information, and outputs the audio signal after the acoustic processing (audio signal 1111).

[0101] Before starting rendering, the spatial information management unit 1201 reads metadata of the input signal, detects rendering items such as objects and sounds defined in the spatial information, and transmits them to the rendering unit 1203. After starting rendering, the spatial information management unit 1201 grasps changes over time in the spatial information and the listener's position, updates and manages the spatial information, and transmits the updated spatial information to the rendering unit 1203.

[0102] The rendering unit 1203 generates and outputs an audio signal to which acoustic processing has been applied, based on the audio signal included in the input data 1113 and the spatial information received from the spatial information management unit 1201 .

[0103] The spatial information update process and the audio signal output process with added acoustic processing may be executed in the same thread. Furthermore, the spatial information management unit 1201 and the rendering unit 1203 may each allocate their processes to independent threads. When the spatial information management unit 1201 and the rendering unit 1203 execute the spatial information update process and the audio signal output process with added acoustic processing in different threads, they may set the thread startup frequency individually, or may execute the processes in parallel.

[0104] When the spatial information management unit 1201 and the rendering unit 1203 execute processes in different independent threads, it is possible to allocate computing resources preferentially to the rendering unit 1203. This makes it possible to safely execute sound output processing in which even the slightest delay is unacceptable, for example, in which a delay of one sample (0.02 msec) would cause a popping noise.

[0105] In this case, the allocation of computational resources to the spatial information management unit 1201 is limited. However, because updating of spatial information is a process that occurs less frequently than output processing of audio signals (for example, a process such as updating the direction of the listener's face), it does not necessarily have to be performed instantaneously like output processing of audio signals. Therefore, even if the allocation of computational resources is limited, it does not have a significant impact on acoustic quality.

[0106] The spatial information may be updated periodically at preset times or intervals, or when preset conditions are met. The spatial information may also be updated manually by a listener or a sound space manager, or may be updated in response to a change in an external system.

[0107] For example, the spatial information may be updated when a listener operates a controller to instantly warp the position of his / her avatar or instantly advance or reverse the time. Alternatively, the spatial information may be updated when an administrator of the virtual space suddenly changes the environment of the space. In these cases, the thread for updating the spatial information managed by the spatial information management unit 1201 may be started as a one-off interrupt process in addition to being started periodically.

[0108] The role of the information update thread that executes the spatial information update process is, for example, to update the position or orientation of the listener's avatar placed in the virtual space based on the position or orientation of the VR goggles worn by the listener, and to update the position of objects moving in the virtual space. These tasks are handled within a processing thread that runs relatively infrequently, on the order of several tens of Hz. Processing that reflects the properties of direct sound may be performed in such an infrequently occurring processing thread. This is because the properties of direct sound change less frequently than the frequency with which audio processing frames for audio output occur. Doing so can actually reduce the computational load of the process relatively, and can also avoid the risk of pulsive noise occurring when information is updated at an unnecessarily fast frequency.

[0109] Fig. 4B is a block diagram showing another example of the configuration of a decoder. Specifically, Fig. 4B shows the configuration of a decoder 1210, which is another example of the decoder 1112 in Fig. 3B or 3D.

[0110] Figure 4B differs from Figure 4A in that the input data 1113 includes an unencoded audio signal rather than encoded audio data. The input data 1113 includes a bitstream including metadata and an audio signal.

[0111] The spatial information management unit 1211 is the same as the spatial information management unit 1201 in FIG. 4A, and therefore a description thereof will be omitted.

[0112] The rendering unit 1213 is the same as the rendering unit 1203 in FIG. 4A, and therefore a description thereof will be omitted.

[0113] The decoders 1112, 1200, and 1210 may be expressed as audio processing units that perform audio processing. The decoding devices 1110 and 1130 may be the audio signal processing devices 1001, and may be expressed as audio processing devices.

[0114] (Physical configuration of audio signal processing device) Fig. 5 is a diagram showing an example of the physical configuration of the audio signal processing device 1001. Note that the audio signal processing device 1001 in Fig. 5 may be the decoding device 1110 in Fig. 3B or the decoding device 1130 in Fig. 3D. The multiple components shown in Fig. 3B or Fig. 3D may be implemented by the multiple components shown in Fig. 5. Furthermore, part of the configuration described here may be provided in the audio presentation device 1002.

[0115] The audio signal processing device 1001 in FIG. 5 includes a processor 1402 , a memory 1404 , a communication IF (Interface) 1403 , a sensor 1405 , and a speaker 1401 .

[0116] The processor 1402 is, for example, a CPU, a DSP (Digital Signal Processor), or a GPU (Graphics Processing Unit). The CPU, DSP, or GPU may perform the acoustic processing or decoding processing of the present disclosure by executing a program stored in the memory 1404. The processor 1402 is, for example, a circuit that performs information processing. The processor 1402 may also be a dedicated circuit that performs signal processing on audio signals, including the acoustic processing of the present disclosure.

[0117] The memory 1404 is configured, for example, with a RAM (Random Access Memory) or a ROM (Read Only Memory). The memory 1404 may include a magnetic recording medium such as a hard disk or a semiconductor memory such as an SSD. The memory 1404 may also be an internal memory incorporated in the CPU or GPU. The memory 1404 may also store spatial information managed by the spatial information management units 1201 and 1211, and threshold data, which will be described later.

[0118] The communication IF 1403 is a communication module compatible with a communication method such as Bluetooth (registered trademark) or WIGIG (registered trademark). The audio signal processing device 1001 communicates with another communication device via the communication IF 1403, for example, to acquire a bitstream to be decoded. The acquired bitstream is stored in the memory 1404, for example.

[0119] The communication IF 1403 is configured with, for example, a signal processing circuit and an antenna corresponding to a communication method. The communication method is not limited to Bluetooth (registered trademark) and WIGIG (registered trademark), but may also be LTE (Long Term Evolution), NR (New Radio), Wi-Fi (registered trademark), or the like.

[0120] Furthermore, the communication method is not limited to the wireless communication method described above, but may be a wired communication method such as Ethernet (registered trademark), USB (Universal Serial Bus), or HDMI (registered trademark) (High-Definition Multimedia Interface).

[0121] The sensor 1405 performs sensing to estimate the position and orientation of the listener. Specifically, the sensor 1405 estimates the position and / or orientation of the listener based on one or more detection results of the position, orientation, movement, velocity, angular velocity, acceleration, etc. of a part or the whole of the body, and generates position / or orientation information indicating the position and / or orientation of the listener.

[0122] Note that a device external to the audio signal processing device 1001 may be equipped with the sensor 1405. The part of the body may be the listener's head, etc. The position / orientation information may be information indicating the position and / or orientation of the listener in real space, or information indicating a displacement of the position and / or orientation of the listener based on the position and / or orientation of the listener at a predetermined time. Furthermore, the position / or orientation information may be information indicating a position and / or orientation relative to the stereophonic sound reproduction system 1000 or an external device equipped with the sensor 1405.

[0123] The sensor 1405 is, for example, an imaging device such as a camera or a ranging device such as a LiDAR (Laser Imaging Detection and Ranging). The sensor 1405 may capture an image of the listener's head movement and detect the head movement by processing the captured image. Alternatively, the sensor 1405 may be a device that performs position estimation using a wireless signal of any frequency band, such as a millimeter wave.

[0124] Furthermore, the audio signal processing device 1001 may acquire position information from an external device equipped with a sensor 1405 via the communication IF 1403. In this case, the audio signal processing device 1001 may not include the sensor 1405. Here, the external device is, for example, the audio presentation device 1002 described in Fig. 2 or a 3D video playback device worn on the head of a listener. In this case, the sensor 1405 is configured by combining various sensors such as a gyro sensor and an acceleration sensor.

[0125] For example, the sensor 1405 may detect the angular velocity of rotation around at least one of three mutually orthogonal axes in the sound space as the axis of rotation as the speed of movement of the listener's head, or may detect the acceleration of displacement with at least one of the three axes as the direction of displacement.

[0126] For example, the sensor 1405 may detect the amount of rotation about at least one of three mutually orthogonal axes in the sound space as the rotation axis, or the amount of displacement about at least one of the three axes as the displacement direction, as the amount of movement of the listener's head. Specifically, the sensor 1405 detects the 6 DoF positions (x, y, z) and angles (yaw, pitch, roll) as the position of the listener. The sensor 1405 is configured by combining various sensors used for detecting movement, such as a gyro sensor and an acceleration sensor.

[0127] The sensor 1405 may be realized by a camera for detecting the position of the listener, a GPS (Global Positioning System) receiver, or the like. Position information obtained by performing self-position estimation using a LiDAR or the like as the sensor 1405 may also be used. For example, when the stereophonic sound reproduction system 1000 is realized by a smartphone, the sensor 1405 is built into the smartphone.

[0128] The sensor 1405 may also include a temperature sensor such as a thermocouple that detects the temperature of the audio signal processing device 1001. The sensor 1405 may also include a sensor that detects the remaining charge of a battery provided in the audio signal processing device 1001 or a battery connected to the audio signal processing device 1001.

[0129] The speaker 1401 has, for example, a diaphragm, a drive mechanism such as a magnet or a voice coil, and an amplifier, and presents an audio signal after acoustic processing as sound to a listener. The speaker 1401 operates the drive mechanism in response to an audio signal (more specifically, a waveform signal indicating the waveform of the sound) amplified via the amplifier, and the drive mechanism vibrates the diaphragm. In this way, the diaphragm vibrating in response to the audio signal generates sound waves, which propagate through the air to the listener's ears, causing the listener to perceive the sound.

[0130] Here, an example has been given in which the audio signal processing device 1001 is provided with a speaker 1401 and an audio signal after acoustic processing is presented via the speaker 1401, but the means for presenting the audio signal is not limited to the above configuration.

[0131] For example, the audio signal after acoustic processing may be output to an external audio presentation device 1002 connected via a communication module. Communication via the communication module may be wired or wireless. As another example, the audio signal processing device 1001 may have a terminal for outputting an analog audio signal, and a cable for earphones or the like may be connected to the terminal to present the audio signal from the earphones or the like.

[0132] In the above case, the audio presentation device 1002 may be headphones, earphones, a head-mounted display, a neck speaker, a wearable speaker, or the like that are worn on the head or part of the body of the listener. Alternatively, the audio presentation device 1002 may be a surround speaker or the like that is composed of multiple fixed speakers. The audio presentation device 1002 may then reproduce an audio signal.

[0133] (Physical Configuration of Encoding Apparatus) Fig. 6 is a diagram showing an example of the physical configuration of encoding apparatus 1500. Encoding apparatus 1500 in Fig. 6 may be encoding apparatus 1100 in Fig. 3A or encoding apparatus 1120 in Fig. 3C, and multiple components shown in Fig. 3A or 3C may be implemented by multiple components shown in Fig. 6.

[0134] The encoding device 1500 in FIG. 6 includes a processor 1501 , a memory 1503 , and a communication IF 1502 .

[0135] The processor 1501 is, for example, a CPU, a DSP, or a GPU. The CPU, DSP, or GPU may perform the encoding process of the present disclosure by executing a program stored in the memory 1503. The processor 1501 is, for example, a circuit that performs information processing. The processor 1501 may be a dedicated circuit that performs signal processing on an audio signal, including the encoding process of the present disclosure.

[0136] The memory 1503 is configured with, for example, a RAM or a ROM. The memory 1503 may include a magnetic recording medium such as a hard disk or a semiconductor memory such as an SSD. The memory 1503 may also be an internal memory incorporated in the CPU or GPU.

[0137] The communication IF 1502 is a communication module compatible with a communication method such as Bluetooth (registered trademark) or WIGIG (registered trademark). The encoding device 1500 communicates with another communication device via the communication IF 1502, for example, and transmits an encoded bitstream.

[0138] The communication IF 1502 is configured with, for example, a signal processing circuit and an antenna corresponding to the communication method. The communication method is not limited to Bluetooth (registered trademark) and WIGIG (registered trademark), but may be LTE, NR, Wi-Fi (registered trademark), or the like. Furthermore, the communication method is not limited to a wireless communication method. The communication method may be a wired communication method such as Ethernet (registered trademark), USB, or HDMI (registered trademark).

[0139] The communication module is composed of, for example, a signal processing circuit and an antenna corresponding to the communication method. In the above example, Bluetooth (registered trademark) or WIGIG (registered trademark) was used as an example of the communication method, but the communication method may also be compatible with communication methods such as LTE (Long Term Evolution), NR (New Radio), or Wi-Fi (registered trademark). Furthermore, the communication IF may be a wired communication method such as Ethernet (registered trademark), USB (Universal Serial Bus), or HDMI (registered trademark) (High-Definition Multimedia Interface) instead of the wireless communication method described above.

[0140] [Configuration of Rendering Unit] Fig. 7 is a block diagram showing an example configuration of the rendering unit 1300. Specifically, Fig. 7 shows an example detailed configuration of the rendering unit 1300 corresponding to the rendering units 1203 and 1213 in Figs. 4A and 4B.

[0141] The rendering unit 1300 is composed of an analysis unit 1301, a selection unit 1302, and a reproduction unit 1303, and applies acoustic processing to sound data contained in an input signal and outputs the result.

[0142] The input signal may be composed of, for example, spatial information, sensor information, and sound data. The input signal may also include a bitstream composed of sound data and metadata (control information), in which case the metadata may include spatial information.

[0143] The spatial information is information about the sound space (three-dimensional sound field) created by the stereophonic sound reproduction system 1000, and is composed of information about objects included in the sound space and information about the listener. Objects include sound source objects that emit sound and act as sound sources, and non-sound-emitting objects that do not emit sound. Sound source objects can also be simply referred to as sound sources.

[0144] A non-sound-emitting object acts as an obstacle object that reflects the sound emitted by a sound source object, but a sound source object may also act as an obstacle object that reflects the sound emitted by another sound source object. Obstacle objects may also be referred to as reflecting objects.

[0145] Information commonly assigned to sound source objects and non-sound generating objects includes position information, shape information, and the rate of attenuation of the volume when the object reflects sound.

[0146] The position information is expressed as coordinate values ​​on three axes, for example, the X-axis, Y-axis, and Z-axis, in Euclidean space, but does not necessarily have to be three-dimensional information. For example, the position information may be two-dimensional information expressed as coordinate values ​​on two axes, the X-axis and the Y-axis. The position information of an object is determined by a representative position of a shape expressed by a mesh or voxels.

[0147] The shape information may include information about the surface material.

[0148] The attenuation rate may be expressed as a real number between 0 and 1, or may be expressed as a negative decibel value. In real space, the volume is not amplified by reflection, so a negative decibel value is set as the attenuation rate, but for example, to create an eerie feeling in an unreal space, an attenuation rate of 1 or more, i.e., a positive decibel value, may be set.

[0149] The attenuation rate may be set to a different value for each of the frequency bands constituting the plurality of frequency bands, or may be set independently for each frequency band. Furthermore, if the attenuation rate is set for each type of material on the object surface, a corresponding attenuation rate value may be used based on information about the surface material.

[0150] The spatial information may also include information indicating whether the object belongs to a living thing, information indicating whether the object is a moving object, etc. If the object is a moving object, the position indicated by the position information may move over time. In this case, information on the changed position or the amount of change is transmitted to the rendering unit 1300.

[0151] The information about the sound source object includes information commonly assigned to the sound source object and the non-sound-producing object, as well as sound data and information necessary for radiating the sound data into the sound space. The sound data is data indicating information about the frequency and intensity of the sound, and is data that expresses the sound perceived by a listener.

[0152] The sound data is typically a PCM signal, but may also be data compressed using an encoding method such as MP3. In this case, the signal must be decoded at least before it reaches the playback unit 1303, so the rendering unit 1300 may include a decoding unit (not shown). Alternatively, the signal may be decoded by the audio data decoder 1202.

[0153] One piece of sound data may be set for one sound source object, or multiple pieces of sound data may be set for one sound source object. Furthermore, identification information for identifying each piece of sound data may be assigned to the sound data, and the information about the sound source object may include the identification information of the sound data.

[0154] The information necessary to radiate sound data into a sound space may include, for example, information on the reference volume used as a standard for playing back sound data, information indicating the properties (also called characteristics) of the sound data, information on the position of the sound source object, and information on the orientation of the sound source object (i.e., information on the directionality of the sound emitted by the sound source object).

[0155] The reference volume information may be, for example, the effective value of the amplitude value of the sound data at the sound source position when the sound data is emitted into the sound space, and may be expressed as a floating-point decibel (db) value.

[0156] For example, a reference volume of 0 db may indicate that sound is emitted into the sound space from the position indicated by the information regarding the position of the sound source object at the same volume as the signal level indicated by the sound data, without increasing or decreasing the volume.Alternatively, a reference volume of -6 db may indicate that sound is emitted into the sound space from the position indicated by the information regarding the position of the sound source object, with the volume of the signal level indicated by the sound data reduced to approximately half.

[0157] The reference volume information may be attached to each piece of sound data, or may be attached to a plurality of pieces of sound data collectively.

[0158] The information indicating the properties of the sound data may be, for example, information relating to the volume of the sound source, and may be information indicating time-series fluctuations in the volume of the sound source.

[0159] For example, if the sound space is a virtual conference room and the sound source is a speaker, the volume transitions intermittently over a short period of time. That is, sound and silence alternate. If the sound space is a concert hall and the sound source is a performer, the volume is maintained for a certain period of time. If the sound space is a battlefield and the sound source is an explosive, the volume of the explosion will increase for a moment and then remain silent or low.

[0160] In this way, the information on the volume of the sound source may include not only information on the loudness of the sound but also information on the transition of the loudness of the sound. Such information may be used as information indicating the properties of the sound data.

[0161] The transition information may be represented by data indicating frequency characteristics in a time series. The transition information may be represented by data indicating the duration of a sound section. The transition information may be represented by data indicating a time series of the duration of a sound section and the duration of a silent section. The transition information may be represented by data listing, in a time series, multiple pairs of durations during which the amplitude of a sound signal can be considered steady (considered to be roughly constant) and the amplitude values ​​of the signal during those durations.

[0162] The transition information may be expressed as data on the duration for which the frequency characteristics of the sound signal can be considered stationary. The transition information may be expressed as data listing, in time series, multiple pairs of durations for which the frequency characteristics of the sound signal can be considered stationary and the frequency characteristics during those periods. The transition information may be expressed, for example, in the form of data indicating the outline of a spectrogram.

[0163] Furthermore, the volume used as the reference for the frequency characteristics may be the reference volume. Information on the reference volume and information indicating the properties of the sound data may be used in a process of calculating the volume of direct sound or reflected sound to be perceived by the listener, or may be used in a process of selecting whether or not to perceive the direct sound or reflected sound. Other examples and usage methods of the information indicating the properties of the sound data will be described later.

[0164] The reflected sound according to this embodiment is an example of an indirect sound. The indirect sound may be a reflected sound, a diffracted sound, or the like. In this embodiment, the description will be given using a reflected sound, which is an example of an indirect sound, but the same processing is performed even if an indirect sound is used instead of a reflected sound.

[0165] Information about the direction of the sound source object (orientation information) is typically expressed using yaw, pitch, and roll. Alternatively, the roll rotation may be omitted, and the direction information of the sound source object may be expressed using azimuth (yaw) and elevation (pitch). The direction information of the sound source object may change over time, and if it changes, it is transmitted to the rendering unit 1300.

[0166] Information about the listener is information about the listener's position and orientation in sound space. The information about the position (position information) is expressed as a position on the XYZ axes in Euclidean space, but it does not necessarily have to be three-dimensional information and may be two-dimensional information. Information about the listener's orientation (orientation information) is typically expressed using yaw, pitch, and roll. Alternatively, the roll rotation may be omitted, and the listener's orientation information may be expressed using azimuth (yaw) and elevation (pitch).

[0167] The position information and orientation information of the listener may change over time, and if so, is transmitted to the rendering unit 1300 .

[0168] The sensor information includes the amount of rotation or displacement detected by a sensor 1405 worn by the listener, as well as the listener's position and orientation. The sensor information is transmitted to the rendering unit 1300, which updates the listener's position and orientation information based on the sensor information. The sensor information may include, for example, position information obtained by a mobile terminal performing self-position estimation using a GPS, a camera, LiDAR, or the like.

[0169] Furthermore, information acquired from outside via a communication module may be detected as sensor information instead of the sensor 1405. Information indicating the temperature of the audio signal processing device 1001 and information indicating the remaining battery capacity may be acquired from the sensor 1405. Furthermore, the computational resources (CPU capacity, memory resources, PC performance, etc.) of the audio signal processing device 1001 or the audio presentation device 1002 may be acquired in real time.

[0170] The analysis unit 1301 analyzes the audio signal contained in the input signal and the spatial information received from the spatial information management units 1201 and 1211, and calculates the information necessary to generate direct sound and reflected sound in the playback unit 1303, as well as the information necessary to select whether or not to generate reflected sound.

[0171] The information required to generate direct sound and reflected sound is, for example, values ​​relating to the path taken by each of the direct sound and the reflected sound to reach the listening position, the time it takes for the sound to reach the listening position, the volume at the time of arrival, etc. The values ​​relating to the path taken by each of the direct sound and the reflected sound to reach the listening position, the time it takes for the sound to reach the listening position, the volume at the time of arrival, etc. are, for example, values ​​indicating the path taken by each of the direct sound and the reflected sound to reach the listening position, the time it takes for the sound to reach the listening position, and the volume at the time of arrival, respectively.

[0172] The information required to select the reflected sound to be output is information indicating the relationship between the direct sound and the reflected sound, such as a value relating to the time difference between the direct sound and the reflected sound, and a value relating to the volume ratio between the direct sound and the reflected sound at the listening position. The value relating to the time difference between the direct sound and the reflected sound and the value relating to the volume ratio between the direct sound and the reflected sound at the listening position are, for example, a value indicating the time difference between the direct sound and the reflected sound and a value indicating the volume ratio between the direct sound and the reflected sound at the listening position, respectively.

[0173] It goes without saying that when the volume is expressed in decibel units on a logarithmic axis (when the volume is expressed in the decibel domain), the volume ratio of two signals is expressed as the difference in decibel values. Specifically, the volume ratio of two signals may be the difference between the amplitude values ​​of each signal when expressed in the decibel domain. This value may be calculated based on an energy value, a power value, or the like. Furthermore, in the decibel domain, this difference may be referred to as a gain difference or simply a gain difference.

[0174] That is, the volume ratio in the present disclosure is essentially a ratio of signal amplitudes, and may be expressed as a sound volume ratio, a volume ratio, an amplitude ratio, a sound level ratio, a sound intensity ratio, a gain ratio, etc. Furthermore, when the unit of volume is decibels, the volume ratio in the present disclosure can of course be rephrased as a volume difference.

[0175] In the present disclosure, the term "volume ratio" typically refers to the gain difference when the volume of two sounds is expressed in decibel units, and in the example embodiments, the threshold data is also typically defined as a gain difference expressed in the decibel domain. However, the volume ratio is not limited to a gain difference in the decibel domain. When a volume ratio expressed in a domain other than the decibel domain is used, the threshold data defined in the decibel domain may be converted into the unit of the calculated volume ratio and used. Alternatively, threshold data defined in each unit may be stored in advance in memory.

[0176] In other words, it is clear that the algorithm in the present disclosure can be applied to solving the problem of the present disclosure even if a ratio of energy values ​​or power values, for example, is used instead of the volume ratio.

[0177] The time difference between the arrival of direct sound and reflected sound is, for example, the time difference between the arrival time of direct sound (arrival time) and the arrival time of reflected sound (arrival time). For simplicity, the time difference between the arrival of direct sound and reflected sound may be referred to as the time difference between direct sound and reflected sound. The time difference between direct sound and reflected sound may be the time difference between the times when the direct sound and reflected sound arrive at the listening position, the difference in the time it takes for the direct sound and reflected sound to arrive at the listening position, or the time difference between the time when the direct sound ends and the time when the reflected sound arrives at the listening position. The calculation method for these values ​​will be described later.

[0178] The selection unit 1302 uses the information calculated by the analysis unit 1301 and the threshold data to select whether or not the reproduction unit 1303 will generate a reflected sound. In other words, the selection unit 1302 determines whether or not to select a reflected sound as a target reflected sound to be generated. In other words, the selection unit 1302 selects which of the multiple reflected sounds the reproduction unit 1303 will generate.

[0179] The threshold data is expressed as a boundary (threshold) between whether the reflected sound is perceived or not, for example, on a graph with the value of the time difference between the direct sound and the reflected sound on the horizontal axis and the volume ratio between the direct sound and the reflected sound on the vertical axis. The threshold data may be expressed as an approximation formula having the value of the time difference between the direct sound and the reflected sound as a variable, or may be expressed as an array having the value of the time difference between the direct sound and the reflected sound as an index and a corresponding threshold.

[0180] The selection unit 1302 selects to generate reflected sound when, for example, the volume ratio between the volume of the direct sound at the time of arrival and the volume of the reflected sound at the time difference between the arrival time of the direct sound and the arrival time of the reflected sound is greater than a threshold value set with reference to threshold data. Note that the volume at the time of arrival means the volume of the sound when it arrives at the listening position.

[0181] The time difference between the arrival time of the direct sound and the arrival time of the reflected sound is, in other words, the difference in the time it takes for the direct sound and the reflected sound to arrive at the listening position. Alternatively, the time difference between the end of the direct sound and the arrival of the reflected sound at the listening position may be used as the time difference between the direct sound and the reflected sound. In this case, threshold data different from the threshold data determined based on the time difference between the arrival time of the direct sound and the arrival time of the reflected sound may be used, or a common threshold data may be used.

[0182] The threshold data may be acquired from the memory 1404 of the audio signal processing device 1001, or may be acquired from an external storage device via a communication module. A method for storing the threshold data and a method for setting the threshold will be described later.

[0183] The reproduction unit 1303 synthesizes the audio signal of the direct sound with the audio signal of the reflected sound that the selection unit 1302 has selected to generate.

[0184] Specifically, the reproduction unit 1303 processes the input audio signal to generate a direct sound based on information about the direct sound arrival time and volume at the time of direct sound arrival calculated by the analysis unit 1301. The reproduction unit 1303 also processes the input audio signal to generate a reflected sound based on information about the reflected sound arrival time and volume at the time of reflected sound arrival for the reflected sound selected by the selection unit 1302. The reproduction unit 1303 then synthesizes and outputs the generated direct sound and reflected sound.

[0185] [Example of Operation of Rendering Unit] Fig. 8 is a flowchart showing an example of operation of the audio signal processing device 1001. Fig. 8 mainly shows processing executed by the rendering unit 1300 of the audio signal processing device 1001.

[0186] In the input signal analysis process (S101 in FIG. 8), the analysis unit 1301 analyzes the input signal input to the audio signal processing device 1001 to detect direct sound and reflected sound that may be generated in the sound space. The reflected sound detected here is a candidate for reflected sound that is selected by the selection unit 1302 as the reflected sound that will ultimately be generated by the reproduction unit 1303. The analysis unit 1301 also analyzes the input signal to calculate information necessary for generating direct sound and reflected sound, and information necessary for selecting the reflected sound to be generated.

[0187] First, the characteristics of each of the direct sound and the reflected sound are calculated. Specifically, the arrival time and volume of each of the direct sound and the reflected sound when they reach the listener are calculated. If multiple objects exist in the sound space as reflecting objects, the characteristics of the reflected sound are calculated for each of the multiple objects.

[0188] The direct sound arrival time (td) is calculated based on the direct sound arrival path (pd). The direct sound arrival path (pd) is a path connecting the position information S (xs, ys, zs) of the sound source object and the position information A1 (xa, ya, za) of the listener. The direct sound arrival time (td) is a value obtained by dividing the length of the path connecting the position information S (xs, ys, zs) and the position information A1 (xa, ya, za) by the speed of sound (approximately 340 m / s).

[0189] For example, the path length (X) can be calculated as (xs-xa)^2 + (ys-ya)^2 + (zs-za)^2)^0.5. The volume attenuates in inverse proportion to the distance. Therefore, if the volume of the sound source object at the position information S(xs, ys, zs) is N and the unit distance is U, the volume of the direct sound (ld) when it arrives can be calculated as ld=N*U / X.

[0190] The volume N at the sound source position may be the reference volume described above.

[0191] The reflected sound arrival time (tr) is calculated based on the reflected sound arrival path (pr), which is a path connecting the position of the sound image of the reflected sound and the position information A1 (xa, ya, za).

[0192] The position of the sound image of the reflected sound may be derived using, for example, the "mirror image method" or "ray tracing method," or any other method for deriving the sound image position. The mirror image method is a method for simulating a sound image by assuming that a mirror image of a wave reflected from a wall in a room exists at a position symmetrical to the sound source with respect to the wall, and that a sound wave is emitted from the position of the mirror image. The ray tracing method is a method for simulating an image (sound image) observed at a certain point by tracing waves that propagate in a straight line, such as light rays or sound rays.

[0193] Fig. 9 is a diagram showing a positional relationship between a listener and an obstacle object where the listener is relatively far away. Fig. 10 is a diagram showing a positional relationship between a listener and an obstacle object where the listener is relatively close. That is, Fig. 9 and Fig. 10 each show an example in which a sound image of a reflected sound is formed at a position symmetrical with respect to the sound source position across a wall. By determining the position of the sound image of the reflected sound on the x, y, and z axes based on this relationship, the reflected sound arrival time can be determined in the same way as the method for calculating the direct sound arrival time.

[0194] The arrival time of a reflected sound (tr) is a value obtained by dividing the length (Y) of the path connecting the position of the sound image of the reflected sound and the position information A1 (xa, ya, za) by the speed of sound (approximately 340 m / sec). The volume attenuates in inverse proportion to the distance. Therefore, if the volume at the sound source position is N, the unit distance is U, and the rate of attenuation of the volume upon reflection is G, the volume at the time of arrival of the reflected sound (lr) can be calculated as lr = N * G * U / Y.

[0195] As explained above, the attenuation factor G may be expressed as a real number between 0 and 1, or may be expressed as a negative decibel value. In this case, the volume of the entire signal is attenuated by G. The attenuation factor may also be set for each frequency band constituting multiple frequency bands. In this case, the analysis unit 1301 multiplies each frequency component of the signal by a specified attenuation factor. In order to reduce the amount of calculation, the analysis unit 1301 may use a representative value or average value of multiple attenuation factors for multiple frequency bands as the overall attenuation factor, and attenuate the volume of the entire signal by that amount.

[0196] Next, the analysis unit 1301 calculates the volume ratio (L), which is the ratio between the volume at the time of arrival of the direct sound (ld) and the volume at the time of arrival of the reflected sound (lr), and the time difference (T) between the direct sound and the reflected sound, which are necessary for selecting the reflected sound to be generated.

[0197] The volume ratio (L), which is the ratio of the volume (ld) when direct sound arrives to the volume (lr) when direct sound arrives, is, for example, the value obtained by dividing the volume (lr) when reflected sound arrives by the volume (ld) when direct sound arrives, and is calculated as follows: L = (N * G * U / Y) / (N * U / X) = G * X / Y. Because the value to be calculated is the volume ratio, the values ​​of N and U may be any predetermined values.

[0198] The time difference (T) between the direct sound and the reflected sound may be, for example, the time difference between the time it takes for the direct sound and the reflected sound to arrive at the listening position. For example, the time difference (T) between the direct sound and the reflected sound to arrive at the listening position can be calculated as T = tr - td.

[0199] The time difference (T) may also be the difference in time between when the direct sound and the reflected sound arrive at the listening position. The time difference (T) may also be the time difference between when the direct sound ends and when the reflected sound arrives at the listening position. In other words, the time difference (T) may be the time difference between when the direct sound ends and when the reflected sound starts at the listening position.

[0200] Next, in the reflected sound selection process (S102 in FIG. 8), the selection unit 1302 selects whether or not the reproduction unit 1303 will generate the reflected sound calculated by the analysis unit 1301. In other words, the selection unit 1302 determines whether or not to select the reflected sound as a target reflected sound to be generated. When there are multiple reflected sounds, the selection unit 1302 selects whether or not to generate each of the reflected sounds. As a result of selecting whether or not to generate each reflected sound, the selection unit 1302 may select one or more target reflected sounds to be generated from among the multiple reflected sounds, or may not select any target reflected sounds to be generated.

[0201] The selection unit 1302 may select reflected sounds to which other processing is to be applied, not limited to the generation processing. For example, the selection unit 1302 may select reflected sounds to which binaural processing is to be applied. Furthermore, the selection unit 1302 basically selects only one or more reflected sounds to be processed. However, the selection unit 1302 may also select only one or more reflected sounds that are not to be processed. Then, processing may be applied to one or more reflected sounds that are not selected.

[0202] For example, the selection of reflected sounds is performed based on the volume ratio (L) and time difference (T) calculated by the analysis unit 1301. By performing the selection process based on the time difference (T) between the direct sound and the reflected sound, it is possible to more appropriately select reflected sounds that have a greater impact on the listener's perception than when the selection process is performed based only on the volume difference between the direct sound and the reflected sound.

[0203] Specifically, the selection of whether to generate reflected sound is made by comparing the volume ratio between the direct sound and the reflected sound, which corresponds to the time difference between the direct sound and the reflected sound, with a preset threshold. The threshold is set with reference to threshold data. The threshold data is an index indicating the boundary between whether a reflected sound relative to the direct sound is perceived by a listener, and is defined as the ratio between the volume of the direct sound (Id) and the volume of the reflected sound (Ir).

[0204] The threshold corresponds to a value expressed by a numerical value or the like determined in correspondence with the time difference (T). The threshold data corresponds to the relationship between the time difference (T) and the threshold, and corresponds to table data or a relational expression used to identify or calculate the threshold for the time difference (T). The format and type of the threshold data are not limited to table data or a relational expression.

[0205] Fig. 11 is a diagram showing the relationship between the time difference between direct sound and reflected sound and a threshold. For example, threshold data of a volume ratio that is predetermined for each value of the time difference between direct sound and reflected sound as shown in Fig. 11 may be referenced. Alternatively, threshold data obtained by interpolation or extrapolation from the threshold data shown in Fig. 11 may be referenced.

[0206] Then, a threshold value for the volume ratio at the time difference (T) calculated by the analysis unit 1301 is identified from the threshold data. Then, the selection unit 1302 determines whether or not to select the reflected sound as a reflected sound to be generated, depending on whether or not the volume ratio (L) between the direct sound and the reflected sound calculated by the analysis unit 1301 exceeds the threshold value.

[0207] By performing selection processing using threshold data of volume ratios that are predetermined for each value of the time difference between direct sound and reflected sound, it is possible to realize selection processing that takes post-masking or precedence effect into consideration. The type, format, storage method, and setting method of threshold data will be described in detail later.

[0208] Next, in the process of generating direct sound and reflected sound (S103 in FIG. 8), the reproduction unit 1303 generates and synthesizes an audio signal of the direct sound and an audio signal of the reflected sound selected by the selection unit 1302 as the reflected sound to be generated.

[0209] The audio signal of the direct sound is generated by applying the direct sound arrival time (td) and the volume at direct sound arrival (ld) calculated by the analysis unit 1301 to the sound data of the sound source object included in the input information. Specifically, the sound data is delayed by the direct sound arrival time (td) and multiplied by the volume at direct sound arrival (ld). The process of delaying the sound data is a process of moving the position of the sound data forward or backward on the time axis. For example, a process of delaying sound data without degrading sound quality, such as that disclosed in Patent Document 2, may be applied.

[0210] The audio signal of the reflected sound is generated by applying the reflected sound arrival time (tr) and the volume at the time of arrival of the reflected sound (lr) calculated by the analysis unit 1301 to the sound data of the sound source object, just like the direct sound.

[0211] However, unlike the volume (ld) of direct sound arrival, the volume (lr) of reflected sound generation is a value to which an attenuation factor G of the volume of reflection is applied. G may be an attenuation factor applied to all frequency bands at once. Alternatively, a reflectance factor may be specified for each predetermined frequency band to reflect the bias of frequency components caused by reflection. In this case, the process of applying the volume (lr) of reflected sound arrival may be performed as a frequency equalizer process, which multiplies each band by an attenuation factor.

[0212] In the above example, the path lengths of the direct sound and the reflected sound candidates as they arrive at the listener are calculated. Furthermore, the arrival times and volumes at the time of arrival are calculated based on the respective path lengths. Then, the reflected sound candidates are selected based on the time difference and volume ratio between them.

[0213] As another example, the selection process may be performed based on the path lengths of the direct sound and the reflected sound as they reach the listener, and the calculation of the arrival times and arrival volumes of the direct sound and the reflected sound, as well as the calculation of the time difference and volume ratio, may be omitted. In this case, a threshold value corresponding to the path length difference may be predetermined for the path length ratio. The selection process may then be performed based on whether the calculated path length ratio is equal to or greater than the threshold value corresponding to the calculated path length difference. This makes it possible to perform the selection process based on the path length difference corresponding to the time difference while reducing the amount of calculation.

[0214] In addition to the path length difference, the value of a parameter indicating the sound propagation velocity or the value of a parameter that affects the sound propagation velocity parameter may also be used.

[0215] (Details of Selection Process) Details of the selection process of whether or not to generate reflected sound will be described.

[0216] The selection of the reflected sound is performed by comparing a threshold value that defines a volume ratio, which is the ratio between the volume of the direct sound when it arrives and the volume of the reflected sound when it arrives, during the time difference (T) between the direct sound and the reflected sound, with the volume ratio (L) calculated by the analysis unit 1301. For example, of the volume ratio threshold values ​​that are predetermined for each value of the time difference between the direct sound and the reflected sound, the volume ratio threshold value for the time difference (T) between the direct sound and the reflected sound calculated by the analysis unit 1301 is referenced. Then, whether or not to select the reflected sound as a reflected sound to be generated is determined depending on whether or not the volume ratio (L) calculated by the analysis unit 1301 exceeds the threshold value.

[0217] The time difference (T) may be, for example, the difference in the time when the direct sound and the reflected sound arrive at the listening position, the time difference between the time it takes for the direct sound and the reflected sound to arrive at the listening position, or the time difference between the time when the direct sound ends and the time when the reflected sound arrives at the listening position. Here, the end time of the direct sound may be calculated by adding the duration of the direct sound to the arrival time of the direct sound.

[0218] The threshold data may be determined based on the minimum time difference at which a listener can perceptually detect a discrepancy between two sounds due to auditory nerve activity or cognitive activity in the brain, more specifically, due to the precedence effect (described below), the temporal masking phenomenon (described below), or a combination thereof. Specific values ​​may be derived from already known research results on the temporal masking effect, the precedence effect, or the echo detection limit, or may be determined through listening experiments assuming application to the virtual space.

[0219] 12A, 12B, and 12C are diagrams showing examples of a method for setting threshold data. As shown in Fig. 12A, 12B, and 12C, the threshold data is represented by a graph in which the horizontal axis represents the time difference between direct sound and reflected sound and the vertical axis represents the volume ratio between direct sound and reflected sound, and the threshold is the boundary (threshold) between whether the reflected sound is perceived or not.

[0220] The threshold data may be expressed by an approximation formula having the time difference between the direct sound and the reflected sound as a variable. Alternatively, the threshold data may be stored in an area of ​​memory 1404 as an array of indexes of the time difference between the direct sound and the reflected sound and thresholds corresponding to the indexes, as shown in FIG.

[0221] Note that when the height of the line parallel to the horizontal axis (minimum audibility limit) in Example 4 of Fig. 12C is used as the threshold, the volume of the reflected sound itself is compared with the threshold, not the volume ratio (L) between the direct sound and the reflected sound. This is because the threshold indicates the volume at the boundary between whether a sound can be perceived by a listener and is a threshold for determining that sounds lower in volume than the threshold will not be reproduced. In other words, the threshold corresponding to the minimum audibility limit is not a threshold for the ratio between the volume of the reflected sound and the volume of the direct sound.

[0222] When the minimum audible limit is used as the threshold, the time difference (T) does not need to be calculated because the threshold is constant regardless of the time difference (T).

[0223] When multiple reflected sounds are generated in the analysis process (S101 in FIG. 8), the selection process may be performed on all reflected sounds, or on only those reflected sounds with high evaluation values ​​based on evaluation values ​​derived for each reflected sound using a preset evaluation method. Here, the evaluation value of a reflected sound corresponds to the perceptual importance of the reflected sound. A high evaluation value corresponds to a large evaluation value, and these expressions may be interchangeable.

[0224] The selection unit 1302 may calculate an evaluation value of the reflected sound using a pre-set evaluation method based on, for example, the volume of the sound source, the visibility of the sound source, the positioning of the sound source, the visibility of the reflecting object (obstacle object), or the geometric relationship between the direct sound and the reflected sound.

[0225] Specifically, the louder the volume of the sound source, the higher the evaluation value may be. Furthermore, in order to match the visual localization with the acoustic localization, the evaluation value may be high when the sound source object or a reflective object (obstacle object) is visible to the listener, or when the localization of the sound source object is high.

[0226] Furthermore, the difference in the arrival angle between the direct sound and the reflected sound and the difference in the arrival time between the direct sound and the reflected sound have a significant impact on the perception of the space, so if the difference in the arrival angle between the direct sound and the reflected sound is large or if the difference in the arrival time between the direct sound and the reflected sound is large, the evaluation value may be high.

[0227] The information on the volume of the sound source may indicate a reference volume defined for each content, a temporal transition of the volume, or both.

[0228] For example, if the virtual space is a virtual conference room and the direct sound is conversation, the volume transitions intermittently over a short period of time. That is, sound and silence alternate. If the virtual space is a concert hall and the direct sound is a musical performance, the volume is maintained for a certain period of time. If the virtual space is a battlefield and the direct sound is an explosion, the volume increases for a moment and then remains silent or low.

[0229] In this way, the volume information of the sound source may include not only information on the reference volume corresponding to the volume setting when the sound is emitted into the virtual space, but also information on the transition of the volume of the sound.

[0230] The transition information may be represented by data indicating frequency characteristics in a time series. The transition information may be represented by data indicating the duration of a sound section. The transition information may be represented by data indicating a time series of the duration of a sound section and the duration of a silent section. The transition information may be represented by data listing, in a time series, multiple pairs of durations during which the amplitude of a sound signal can be considered steady (considered to be roughly constant) and the amplitude values ​​of the signal during those durations.

[0231] The transition information may be expressed as data on the duration for which the frequency characteristics of the sound signal can be considered to be stationary, or may be expressed as data listing, in chronological order, multiple pairs of durations for which the frequency characteristics of the sound signal can be considered to be stationary and the frequency characteristics during those periods.

[0232] Furthermore, efforts to use temporal transitions in the frequency characteristics of signals in acoustic processing of virtual spaces have been widely undertaken in the past (see, for example, Patent Document 1). In light of such prior art, it goes without saying that the above pair may be a pair of a time length during which the frequency characteristics are constant and the frequency characteristics themselves.

[0233] The geometric relationship may be the relationship between the positions of the sound source, the listener, and the reflecting object in the virtual space. These relationships allow the geometric calculation of the path lengths of the direct sound and the reflected sound. Therefore, by utilizing the relationship in which the volume is inversely proportional to the distance, it is possible to calculate the reference volume of the reflected sound relative to the reference volume of the direct sound.

[0234] The reference volume of the reflected sound may be calculated using the reflection coefficient of the reflecting object. A commonly used typical value may also be used as the reflection coefficient. On the other hand, if a special condition exists, such as the reflecting object being covered with a sound-absorbing material, a specially assigned reflection coefficient may be used as the reflection coefficient of the reflecting object.

[0235] The reflected sound may be evaluated based on its volume, which may be calculated from the geometric relationship between the direct sound and the reflected sound and the index assigned to the reflective object, as described above, and may be evaluated by comparing the volume with a predetermined threshold.

[0236] Furthermore, information indicating the temporal transition of the volume of the sound source may be reflected in the evaluation. For example, if the information indicating the temporal transition of the volume of the sound source indicates the duration of a sound section, and the time is within the sound section, the evaluation value of the reflected sound may be maintained as is. On the other hand, if the time is outside the sound section, processing may be performed to reduce or set the evaluation value of the reflected sound to zero even if the reference volume of the reflected sound exceeds the threshold.

[0237] Alternatively, the information indicating the temporal transition of the volume of the sound source may be data listing, in time series, multiple pairs of durations during which the amplitude of a sound signal is considered to be roughly constant and the amplitude values ​​of the signal during those durations. In this case, the reference volume of the reflected sound may be changed in conjunction with changes in the amplitude values ​​in the data to evaluate the reflected sound.

[0238] Furthermore, both information on the reference volume and information on the volume that changes over time may be used as information indicating the volume of the direct sound. For example, after an evaluation value is calculated based on information on the reference volume, the evaluation value may be corrected using information on the volume that changes over time.

[0239] In the evaluation of reflected sounds, all of the above-described methods may be executed, or only some of them may be executed. For example, reflected sounds may be evaluated using a plurality of evaluation methods, or may be evaluated using a single evaluation method.

[0240] When reflected sound is evaluated using multiple evaluation methods, whether or not to select the reflected sound may be determined based on an evaluation value determined comprehensively using the multiple evaluation methods, or may be determined based on the evaluation values ​​for each of the multiple evaluation methods.

[0241] When determining whether to select a reflected sound based on each of a plurality of evaluation methods, the audio signal processing device 1001 may select a sound if all of the plurality of evaluation results based on the plurality of evaluation methods indicate that the sound should be selected. Alternatively, the audio signal processing device 1001 may select a sound if any one of the plurality of evaluation results based on the plurality of evaluation methods indicates that the sound should be selected.

[0242] Furthermore, for example, priorities may be assigned to the first to third evaluation methods. Then, when it is determined that sound should not be selected using the first evaluation method, the audio signal processing device 1001 may ultimately determine that sound should not be selected without depending on the determination results of the second and third evaluation methods. Furthermore, when it is determined that sound should not be selected using one of the second and third evaluation methods but that sound should be selected using the other, the audio signal processing device 1001 may ultimately determine that sound should be selected.

[0243] Furthermore, the selection process and the evaluation process may be performed independently, or only one of them may be performed. Alternatively, the evaluation process may be performed only on reflected sounds that have been determined to be selected in the selection process, and the evaluation process may then re-determine whether or not to select the reflected sounds. Alternatively, the evaluation process may be performed only on reflected sounds that have been determined not to be selected in the selection process, and the evaluation process may then re-determine whether or not to select the reflected sounds.

[0244] The above-described selection process can be interpreted as a process of selecting reflected sounds according to the properties of direct sounds. For example, in the process of selecting reflected sounds according to the properties of direct sounds, a threshold value used for selecting reflected sounds is set or adjusted according to the properties of the direct sounds. Alternatively, an evaluation value used for selecting reflected sounds is calculated based on one or more of the volume of a sound source, the visibility of a sound source, the localization of a sound source, the visibility of a reflecting object (obstacle object), and the geometric relationship between the direct sound and the reflected sound.

[0245] Furthermore, the process of selecting reflected sounds according to the properties of direct sounds is not limited to the process of setting or adjusting a threshold value according to the properties of direct sounds and the process of calculating an evaluation value used to select reflected sounds to be processed, and other processes may be performed. Even when the process of setting or adjusting a threshold value according to the properties of direct sounds or the process of calculating an evaluation value used to select reflected sounds to be processed is performed, the process may be partially changed or new processes may be added.

[0246] Note that setting the threshold value may include adjusting the threshold value, changing the threshold value, and the like.

[0247] [Method of Setting Thresholds] The threshold data used in the selection process may be set with reference to, for example, an echo detection limit based on the already known precedence effect, or a masking threshold based on the post-masking effect.

[0248] The precedence effect is a phenomenon in which, when sounds are heard from two locations, the one heard first is perceived as the source of the sound. If two short sounds merge and sound like a single sound, the location where the entire sound is heard (localization) is largely determined by the location of the first sound. The echo detection limit is a phenomenon caused by the precedence effect, and is the minimum time difference at which a listener can perceive a discrepancy between two sounds.

[0249] 12C, the horizontal axis corresponds to the arrival time of the reflected sound (echo), specifically, the delay time from the arrival time of the direct sound to the arrival time of the reflected sound, and the vertical axis corresponds to the volume ratio of the detectable reflected sound to the direct sound, specifically, the threshold value for whether the reflected sound arriving with a delay is detectable.

[0250] Fig. 13 is a diagram showing an example of a method for setting a threshold. The horizontal axis in Fig. 13 corresponds to the arrival time of the reflected sound, specifically, the time difference (T) between the direct sound and the reflected sound. The vertical axis in Fig. 13 corresponds to the volume of the reflected sound. Specifically, the vertical axis in Fig. 13 may correspond to the volume of the reflected sound determined relatively to the volume of the direct sound (volume ratio), or may correspond to the volume of the reflected sound determined absolutely regardless of the volume of the direct sound.

[0251] For example, when the listener and the obstacle object are relatively far apart as shown in Fig. 9, the arrival time of the reflected sound is delayed, and the threshold value is set low, as shown in C of Fig. 13. As a result, reflected sound is generated in the case of Fig. 9. On the other hand, when the listener and the obstacle object are relatively close, as shown in Fig. 10, the arrival time of the reflected sound is earlier than in the case of Fig. 9, and the threshold value is set high, as shown in B of Fig. 13. As a result, reflected sound is not generated in the case of Fig. 10.

[0252] The threshold data may also be stored in the memory 1404, retrieved from the memory 1404 during the selection process, and used in the selection process.

[0253] 14 is a flowchart showing an example of the selection process. First, the selection unit 1302 specifies the reflected sound detected by the analysis unit 1301 (S201). Then, the selection unit 1302 detects the volume ratio (L) between the direct sound and the reflected sound and the time difference (T) between the direct sound and the reflected sound (S202 and S203).

[0254] The time difference (T) may be, for example, the time difference between the time it takes for a direct sound and a reflected sound to arrive at the listening position, the time difference between the arrival time of the direct sound and the arrival time of the reflected sound, or the time difference between the time when the direct sound ends and the time when the reflected sound arrives at the listening position. Here, an example based on the time difference between the arrival time of the direct sound and the arrival time of the reflected sound will be described.

[0255] Specifically, the selection unit 1302 calculates the difference between the path length of the direct sound and the path length of the reflected sound from the position information of the sound source object and the listener, and the position information and shape information of the obstacle object.The selection unit 1302 then divides this difference in length by the speed of sound to detect the time difference (T) between the time when the direct sound arrives at the listener's position and the time when the reflected sound arrives at the listener's position.

[0256] The volume of the sound reaching the listener attenuates in proportion to the distance to the listener (inversely proportional to the distance) relative to the volume of the sound source. Therefore, the volume of the direct sound is obtained by dividing the volume of the sound source by the path length of the direct sound. The volume of the reflected sound is obtained by dividing the volume of the sound source by the path length of the reflected sound and then multiplying the result by the attenuation rate assigned to the virtual obstacle object. The selection unit 1302 detects the volume ratio by calculating the ratio between these volumes.

[0257] The selection unit 1302 also uses the threshold data to identify a threshold corresponding to the time difference (T) (S204), and determines whether the detected volume ratio (L) is equal to or greater than the threshold (S205).

[0258] If the volume ratio (L) is equal to or greater than the threshold (Yes in S205), the selection unit 1302 selects the reflected sound as the reflected sound to be generated (S206). If the volume ratio (L) is smaller than the threshold (No in S205), the selection unit 1302 does not select the reflected sound as the reflected sound to be generated (S207). That is, in this case, the selection unit 1302 determines that the reflected sound is not to be generated.

[0259] Thereafter, the selection unit 1302 determines whether or not there is an unspecified reflected sound (S208). If there is an unspecified reflected sound (Yes in S208), the selection unit 1302 repeats the above-described processing (S201 to S207). If there is no unspecified reflected sound (No in S208), the selection unit 1302 ends the processing.

[0260] This selection process may be performed on all reflected sounds generated in the analysis process, or may be performed only on the reflected sounds with high evaluation values ​​described above.

[0261] [Details of Threshold Storage Method] The threshold data according to this embodiment is stored in the memory 1404 of the audio signal processing device 1001. The stored threshold data may be in any format and of any type. When multiple formats and types of thresholds are stored, the selection process may determine which format and type of threshold to use in the selection process of the reflected sounds. A method for determining which threshold data to use in the selection process will be described later.

[0262] Furthermore, threshold data in a plurality of formats and of a plurality of types may be stored in combination. The combined threshold data may be read from the spatial information management units 1201 and 1211, and a threshold to be used in the selection process may be set. The threshold data stored in the memory 1404 may be stored in the spatial information management units 1201 and 1211.

[0263] The threshold data may be stored as thresholds at each time difference, for example, as shown in [Example 1] and [Example 2] of FIG. 12C.

[0264] Furthermore, the threshold data may be stored as table data in which thresholds and time differences (T) are associated with each other, as shown in FIG. 11 . That is, the threshold data may be stored as table data having the time difference (T) as an index. Of course, the thresholds shown in FIG. 11 are merely an example, and the thresholds are not limited to the example of FIG. 11 . Furthermore, instead of storing the thresholds themselves, the thresholds may be approximated by a function having the time difference (T) as a variable, and the coefficients of the function may be stored. Furthermore, a combination of multiple approximation formulas may be stored.

[0265] For example, the threshold data may be expressed by the following formula, where the time difference (T) is timeDiff and the threshold is gainThresh.

[0266]

[0267] The threshold is defined only within the time range in which the precedence effect is expected to occur. For time differences outside this time range (values ​​of 1 ms or less or 40 ms or more in the above formula), the determination using gainThresh may not be performed, and the determination may be performed only based on a threshold indicating the minimum volume reproduced in the virtual space, as described below.

[0268] Experiments conducted by the present inventors have revealed that it is desirable to approximate the threshold with an upwardly convex function in the time range in which the precedence effect is believed to occur. The above formula is an example of an approximation formula generated based on the experiments.

[0269] The memory 1404 may store information regarding a relational expression showing the relationship between the time difference (T) and the threshold value. That is, an expression having the time difference (T) as a variable may be stored. The threshold value of each time difference (T) may be approximated by a straight line or a curve, and parameters indicating the geometric shape of the line or curve may be stored. For example, if the geometric shape is a straight line, the starting point and slope for expressing the straight line may be stored.

[0270] Furthermore, the type and format of threshold data may be determined and stored for each characteristic of the direct sound. Furthermore, parameters for adjusting the threshold according to the characteristic of the direct sound and using it in the selection process may be stored. The process of adjusting the threshold according to the characteristic of the direct sound and using it in the selection process will be described later as a modified example of the threshold setting method.

[0271] As an example of storing a combination of multiple types of threshold data, the larger of the masking threshold and the echo detection limit threshold may be stored for each time difference (T) as shown in [Example 3] of Fig. 12C. Alternatively, the larger of the minimum volume reproduced in the virtual space and the echo detection limit threshold may be stored for each time difference (T) as shown in [Example 4] of Fig. 12C.

[0272] The combination of multiple types of threshold data is not limited to this. For example, maximum value information for each time difference (T) in multiple types of threshold data may be stored.

[0273] In the above description, the information about the threshold value has a one-dimensional index representing the time. The information about the threshold value may also have a two-dimensional or three-dimensional index including a variable relating to the direction of arrival.

[0274] 15 is a diagram showing the relationship between the direction of a direct sound, the direction of a reflected sound, the time difference, and a threshold value. For example, as shown in FIG. 15, threshold values ​​calculated in advance according to the relationship between the direction of a direct sound (θ), the direction of a reflected sound (γ), the time difference (T), and the volume ratio (L) may be stored.

[0275] The direction of direct sound (θ) corresponds to the angle of the direction from which the direct sound arrives relative to the listener. The direction of reflected sound (γ) corresponds to the angle of the direction from which the reflected sound arrives relative to the listener. Here, the direction the listener is facing is defined as 0 degrees. The time difference (T) corresponds to the difference between the time when the direct sound arrives at the listening position and the time when the reflected sound arrives. The volume ratio (L) corresponds to the volume ratio between the volume when the direct sound arrives and the volume when the reflected sound arrives.

[0276] Of course, the thresholds shown in Fig. 15 are merely an example, and the thresholds are not limited to the example of Fig. 15. Also, Fig. 15 mainly illustrates thresholds when the angle (θ) of the arrival direction of the direct sound is 0 degrees. However, thresholds when the arrival direction (θ) of the direct sound is other than 0 degrees are also stored in memory 1404.

[0277] In the above description, the thresholds are stored in an array having the direction of the direct sound (θ) (more specifically, the angle (θ) of the direction from which the direct sound arrives) and the direction of the reflected sound (γ) (more specifically, the angle (γ) of the direction from which the reflected sound arrives) as independent variables or indexes. However, the angle (θ) of the direction from which the direct sound arrives and the angle (γ) of the direction from which the reflected sound arrives do not have to be used as independent variables.

[0278] For example, the angle difference between the angle (θ) of the arrival direction of the direct sound and the angle (γ) of the arrival direction of the reflected sound may be used. This angle difference corresponds to the angle between the arrival direction of the direct sound and the arrival direction of the reflected sound, and may be expressed as the arrival angle between the direct sound and the reflected sound.

[0279] Fig. 16 is a diagram showing the relationship between the angle difference, the time difference, and the threshold. For example, a threshold calculated in advance using the angle difference (Φ) between the angle (θ) of the arrival direction of the direct sound and the angle (γ) of the arrival direction of the reflected sound as a variable may be stored as in the example shown in Fig. 16. Of course, the threshold shown in Fig. 16 is just an example, and the threshold is not limited to the example of Fig. 16.

[0280] 16, it is possible to reduce the number of variables used to derive thresholds, which in turn makes it possible to reduce the number of thresholds stored in memory 1404. Therefore, it is possible to reduce the amount of data stored in memory 1404.

[0281] In addition, when the angle difference (Φ) between the angle (θ) of the arrival direction of the direct sound and the angle (γ) of the arrival direction of the reflected sound is used, the threshold data may be stored in a two-dimensional array. In addition, in the selection process, the difference between the angle (θ) of the arrival direction of the direct sound and the angle (γ) of the arrival direction of the reflected sound may be calculated using a three-dimensional array.

[0282] A method for selecting reflected sounds using a threshold value according to the direction of arrival will be described later.

[0283] 12A , 12B, and 12C , multiple formats and multiple types of thresholds may be stored in the spatial information management units 1201 and 1211. Then, it may be determined which format and which type of threshold to use in the reflected sound selection process from the multiple formats and multiple types of thresholds. Specifically, as shown in example 3 of FIG. 12C , the highest threshold may be adopted for the time difference (T) corresponding to the arrival time of the reflected sound.

[0284] Furthermore, as shown in Example 4, a masking threshold, an echo detection threshold, and a threshold indicating the minimum volume to be reproduced in the virtual space may be stored, and the highest threshold may be adopted for the time difference (T) corresponding to the arrival time of the reflected sound.

[0285] [Second Modification of Threshold Setting Method] As another example of the threshold setting method, a method of setting a threshold depending on the properties of the direct sound will be described.

[0286] Fig. 17 is a block diagram showing another example configuration of the rendering unit 1300 shown in Fig. 7. The rendering unit 1300 in Fig. 17 differs from the rendering unit 1300 in Fig. 7 in that it includes a threshold adjustment unit 1304. The description of the components other than the threshold adjustment unit 1304 is omitted because they are the same as those described in Fig. 7.

[0287] The threshold adjustment unit 1304 selects a threshold to be used by the selection unit 1302 from the threshold data based on information indicating the properties of the audio signal. Alternatively, the threshold adjustment unit 1304 may adjust the threshold included in the threshold data based on information indicating the properties of the audio signal.

[0288] The information indicating the properties of the audio signal may be included in the input signal. Then, the threshold adjustment unit 1304 may acquire the information indicating the properties of the audio signal from the input signal. Alternatively, the analysis unit 1301 may derive the properties of the audio signal by analyzing the audio signal included in the received input signal, and output the information indicating the properties of the audio signal to the threshold adjustment unit 1304.

[0289] The information indicating the characteristics of the audio signal may be obtained before the rendering process begins, or may be obtained each time the rendering process is performed.

[0290] Furthermore, the threshold adjustment unit 1304 does not have to be included in the audio signal processing device 1001, and another communication device may fulfill the role of the threshold adjustment unit 1304. In this case, the analysis unit 1301 or the selection unit 1302 may acquire information indicating the properties of the audio signal, threshold data according to the properties, or information for adjusting the threshold data according to the properties from the other communication device via the communication IF 1403.

[0291] Fig. 18 is a flowchart showing another example of the selection process. Fig. 19 is a flowchart showing yet another example of the selection process. In Fig. 18 and Fig. 19, a threshold is set according to the properties of the direct sound. Specifically, in Fig. 18, the threshold adjustment unit 1304 specifies a threshold from threshold data based on the time difference (T) and the properties of the audio signal. In Fig. 19, the threshold adjustment unit 1304 adjusts the threshold specified from the threshold data based on the time difference (T) based on the properties of the audio signal.

[0292] The operation of each example will be described below, with the explanation of the processes common to the example in FIG.

[0293] First, an example of processing shown in Fig. 18 will be described. Here, threshold data for each property of direct sound is stored in advance in memory 1404. As a result, multiple threshold data corresponding to multiple properties are stored in advance in memory 1404. Then, the threshold adjustment unit 1304 identifies threshold data to be used in the selection processing of reflected sounds from the multiple threshold data.

[0294] For example, the threshold adjustment unit 1304 acquires the characteristics of the direct sound based on the input signal (S211). The threshold adjustment unit 1304 may acquire the characteristics of the direct sound associated with the input signal. Then, the threshold adjustment unit 1304 identifies a threshold corresponding to the time difference (T) and the characteristics of the direct sound (S212).

[0295] As shown in FIG. 19, the threshold value adjusting unit 1304 may adjust the threshold value specified by the selecting unit 1302 based on the properties of the direct sound (S221).

[0296] In either case, the input signal may include information indicating the characteristics of the audio signal, information for adjusting the threshold in accordance with the characteristics of the audio signal, or both of these, and the threshold adjustment unit 1304 may adjust the threshold using one or both of these.

[0297] Furthermore, the information indicating the properties of the audio signal, the information for adjusting the threshold, or both may be transmitted in an input signal other than the input signal containing the audio signal. In this case, the input signal containing the audio signal may include information associating the other input signal with the input signal, or the information associating the other input signal with the input signal may be stored in memory 1404 together with information regarding the threshold.

[0298] In the examples of Figures 18 and 19, the threshold value used to select the reflected sound is set according to the properties of the direct sound, i.e., the properties of the audio signal. Threshold data set in advance for each property may be used, as in Figure 18, or the threshold value may be adjusted according to the properties of the audio signal, as in Figure 19. Furthermore, the parameters of the threshold data may be adjusted according to the properties of the audio signal.

[0299] The operation performed by the threshold adjustment unit 1304 may be performed by the analysis unit 1301 or the selection unit 1302. For example, the analysis unit 1301 may acquire the properties of the audio signal. Alternatively, the selection unit 1302 may set the threshold according to the properties of the audio signal.

[0300] Next, the relationship between the characteristics of the audio signal and the threshold will be described.

[0301] Two short sounds that arrive consecutively at a listener's ears will be heard as a single sound if the time interval between them is sufficiently short. This phenomenon is called the precedence effect. It is known that the precedence effect occurs only for discontinuous, i.e., transient, sounds (Non-Patent Document 1). Therefore, when an audio signal represents a stationary sound, the echo detection threshold may be set lower than when the audio signal represents a non-stationary sound.

[0302] That is, in accordance with the characteristics of such precedence effect, for example, if the direct sound is a steady sound, the threshold value is set to be small. Also, the higher the steadyness, the smaller the threshold value may be set.

[0303] An example of processing when the nature of the audio signal is stationary will be described. First, the threshold adjustment unit 1304 or the analysis unit 1301 determines stationarity based on the amount of fluctuation in the frequency components of the audio signal over time. For example, if the amount of fluctuation is small, the stationarity is determined to be high. Conversely, if the amount of fluctuation is large, the stationarity is determined to be low. As a result of the determination, a flag indicating the level of stationarity may be set, or a parameter indicating stationarity may be set according to the amount of fluctuation.

[0304] Next, the threshold adjustment unit 1304 may adjust the threshold data or threshold based on information indicating stationarity, such as a flag or parameter indicating the stationarity of the audio signal, and set the adjusted threshold data or threshold as the threshold data or threshold to be used in the selection unit 1302.

[0305] Alternatively, parameters for setting threshold data according to information indicating the continuity of the direct sound may be stored in advance in the memory 1404. In this case, the threshold adjustment unit 1304 may determine the continuity of the audio signal, and set threshold data used for selecting reflected sounds based on the information indicating the continuity and the parameters.

[0306] Alternatively, multiple parameters of the threshold data may be stored in advance in memory 1404 in correspondence with multiple patterns of the continuity of the direct sound. In this case, threshold adjustment unit 1304 may determine the continuity of the audio signal, select parameters of the threshold data based on the pattern of the continuity of the direct sound, and set threshold data to be used for selecting reflected sounds based on the parameters of the threshold data.

[0307] The constancy of an audio signal may be determined based on the amount of fluctuation in the frequency components of the audio signal each time the audio signal is input.

[0308] Alternatively, the continuity of the audio signal may be determined based on information indicating the continuity that is pre-linked to the audio signal. That is, the information indicating the continuity of the audio signal may be pre-linked to the audio signal and stored in the memory 1404. The analysis unit 1301 may acquire the information indicating the continuity that is pre-linked to the audio signal every time an audio signal is input. Then, the threshold adjustment unit 1304 may adjust the threshold based on the information indicating the continuity that is pre-linked to the audio signal.

[0309] As another example of how the threshold may be set depending on the nature of the audio signal, the echo detection limit may be set to a shorter range if the audio signal represents a short sound (such as a click) than if the audio signal represents a long sound. This process is based on the properties of the precedence effect.

[0310] It is known that due to the precedence effect, two short sounds that arrive consecutively at a listener's ears are perceived as a single sound if the time interval between them is sufficiently short. The upper limit of this time interval depends on the duration of the sounds. For example, the upper limit of this time interval is about 5 ms for a click sound, but can be as long as 40 ms for complex sounds such as human voices or music (Non-Patent Document 1).

[0311] According to the characteristics of such precedence effect, for example, if the duration of the direct sound is short, a short threshold value is set. Also, the shorter the duration of the direct sound, the shorter the threshold value is set.

[0312] Setting a short threshold value means that a threshold value corresponding to an echo detection limit based on the characteristics of the precedence effect is set within a range where the time difference (T) between the direct sound and the reflected sound is small. Outside this range, a threshold value corresponding to an echo detection limit based on the characteristics of the precedence effect is not set. In other words, outside this range, the threshold value is small. Therefore, setting a short threshold value for a short sound can correspond to setting a small threshold value for a short sound.

[0313] As another example of setting the threshold depending on the characteristics of the direct sound, if the direct sound is an intermittent sound (such as speech), the threshold may be set lower than if the direct sound is a continuous sound (such as music).

[0314] For example, when the direct sound corresponds to speech, sound and silence portions are repeated, and only the post-masking effect occurs in the silence portions. On the other hand, when the direct sound is a continuous sound such as music content, both the post-masking effect and the simultaneous masking effect due to the sound occurring at that time occur. Therefore, the overall masking effect is higher in the case of music than in the case of speech.

[0315] According to the characteristics of the masking effect as described above, the threshold may be set higher for music, etc. than for speech, etc. Conversely, the threshold may be set lower for speech, etc. than for music, etc. In other words, if the direct sound has many intermittent parts, the threshold may be set lower.

[0316] As described above, the information indicating the properties of the direct sound may be information indicating the constancy, intermittency, duration, etc. of the direct sound. Furthermore, the information indicating the properties of the direct sound may be any combination of these. Furthermore, the information indicating the properties of the direct sound may be information indicating the time variation of any of these, or information indicating the time variation of any combination of these. In other words, the information indicating the properties of the direct sound may be information indicating the time variation of the direct sound.

[0317] For example, as described in the description of the stationarity determination, the information indicating the properties of the direct sound may be time-series data of frequency characteristics, where the frequency characteristics may be expressed in a commonly used format such as a gain value for each frequency band, a Fourier series for a time-domain signal, or an LPC coefficient or cepstrum coefficient for determining a frequency envelope.

[0318] Furthermore, the information indicating the properties of the direct sound may be information indicating the intermittency of the direct sound, which lists in chronological order a plurality of pairs of durations during which the amplitude of a signal is steady and the amplitude values ​​of the signal during those durations (an outline of the amplitude envelope). Here, the amplitude values ​​may be expressed as a ratio to a reference volume.

[0319] Furthermore, the information indicating the properties of the direct sound may be information regarding the frequency characteristics of the direct sound. For example, the information indicating the properties of the direct sound may be information indicating the constancy of the frequency characteristics of the direct sound. Specifically, the information indicating the properties of the direct sound may be information (approximate spectrogram shapes) listing in time series multiple pairs of durations during which the frequency characteristics are small and the frequency characteristics of the signal during those durations. Here, the volume used as a reference for the frequency characteristics may be the reference volume.

[0320] For example, the information indicating the time variation of the direct sound is information indicating the envelope of the direct sound. The information indicating the time variation of the direct sound may be used when the "minimum audible limit" described in [Example 4] of Fig. 12C is the threshold. The signal to be compared with the minimum audible limit is the volume of the reflected sound.

[0321] The volume of reflected sound is obtained by geometric calculation using information on the positions of the sound source, listener, and reflecting object. Specifically, the reference volume of the reflected sound relative to the reference volume of the sound source is obtained. By increasing or decreasing the reference volume of the reflected sound using information on the transition of the sound source's loudness as information indicating the properties of the direct sound, it is possible to accurately determine the volume of the reflected sound from moment to moment. This is because fluctuations in the volume of the sound source are reflected in fluctuations in the volume of the reflected sound.

[0322] After adjusting the volume of the reflected sound, the volume of the reflected sound is compared with a threshold value, thereby making it possible to more accurately select the reflected sound that is required for auditory perception.

[0323] Of course, it goes without saying that the same result can be obtained by adjusting the threshold based on the inverse of the information on the transition in loudness of the sound source, without adjusting the reference volume of the reflected sound, and then comparing the adjusted threshold with the reference volume of the reflected sound. In other words, the reference volume of the reflected sound may be adjusted using the information on the transition in loudness of the sound source, or the threshold may be adjusted using the information on the transition in loudness of the sound source. Adjustment of the reference volume of the reflected sound and adjustment of the threshold correspond to each other.

[0324] Depending on the composition of the surface of an object that reflects sound, the sound reflectance (the rate at which sound decays due to reflection) varies for each frequency band. Therefore, as will be described later, a sound reflectance (decay rate) may be associated with each sound reflecting object for each frequency band. Using such reflectance information and spectrogram information, it is possible to more accurately determine whether or not to select the reflected sound. For example, the following processing is performed.

[0325] Specifically, for example, spectrogram information may indicate that high frequency components are more prevalent than low frequency components in a certain time interval, and sound reflectance information may indicate that high frequency components have significantly lower reflectance than low frequency components.

[0326] In this case, even if the amplitude of the sound source signal on the time axis is large, the volume of the reflected sound obtained by multiplying the frequency components indicated by the spectrogram information by the attenuation rate for each frequency band indicated by the reflectivity information will be small, and the reflected sound may not be selected.

[0327] As described above, the information indicating the characteristics of the direct sound may be information indicating a time variation of the direct sound. For example, the information indicating the characteristics of the direct sound may indicate a value obtained by analyzing the direct sound for a predetermined time length.

[0328] Specifically, the information indicating the properties of the direct sound may be information obtained by calculating the average energy or average amplitude of the direct sound for each predetermined time length. Alternatively, the information indicating the properties of the direct sound may be information obtained by calculating the energy or average amplitude of the direct sound for each short-term analysis length and calculating a weighted average of the energy or average amplitude for each long-term analysis length longer than the short-term analysis length.

[0329] More specifically, for example, the information indicating the time variation of the direct sound may be information obtained by calculating the energy or average amplitude of the direct sound for each predetermined short time length (for example, 5 ms; hereinafter, frames of this time length will be referred to as analysis frames). Furthermore, the information indicating the time variation of the direct sound may be information represented by a weighted average of the energy or average amplitude calculated for the past N-1 analysis frames.

[0330] If the energy of the n-th analysis frame is expressed as E(n), information I(n) indicating the properties of the direct sound can be calculated according to the following equation.

[0331]

[0332] Here, the parameter a(i) represents a weighting coefficient. Generally, a(i) is set so that a(i)≧0 and the sum of a(i) is 1. However, the method for setting a(i) is not limited to this.

[0333] Note that information I(n) indicating the properties of the direct sound is calculated every 5 ms that the direct sound is captured. In other words, it is possible to calculate the time variation of information I(n) indicating the properties of the direct sound with low delay. Therefore, this method is suitable for application requiring real-time performance.

[0334] Furthermore, information I(n) indicating the properties of the direct sound may be calculated according to the following formula:

[0335]

[0336] Here, the parameter b(i) represents a weighting coefficient. Generally, b(i) is set so that b(i)≧0 and the sum of b(i) is 1. However, the method for setting b(i) is not limited to this.

[0337] In this equation, information I(n) indicating the properties of the direct sound is calculated recursively, making it possible to calculate the average energy over a long time period with a small amount of calculation.

[0338] The above formulas 1 and 2 can be considered as filters in which E(n) is the input signal and I(n) is the output signal. In this case, formula 1 is a moving average (MA) model filter, and formula 2 is an autoregressive (AR) model filter, both of which have low-pass filter characteristics. Alternatively, an ARMA model filter, which is a combination of both, may be used.

[0339] Note that the method of deriving the information indicating the time variation of the direct sound is not limited to the above-described formula or filter, and other known methods may be used. As described above, the information indicating the time variation of the direct sound indicates a value obtained by analyzing the direct sound for a predetermined time length. The direct sound may be analyzed from a perspective other than average energy.

[0340] As described above, the information indicating the properties of the direct sound may be information related to the frequency characteristics of the direct sound. The information related to the frequency characteristics of the direct sound may be information calculated using the frequency characteristics of the direct sound. For example, the information related to the frequency characteristics of the direct sound may be information obtained as the average energy of the low-frequency components by averaging the low-frequency components of the direct sound over a predetermined analysis length.

[0341] Specifically, a low-pass filter is applied to the direct sound included in the analysis frame length to obtain the low-frequency components of the direct sound. Information indicating the properties of the direct sound is derived from the energy or average amplitude of the low-frequency components, as in the above-described Equation 1.

[0342] If the energy of the low frequency component of the n-th analysis frame is expressed as EL(n), the information I(n) indicating the properties of the direct sound can be calculated according to the following equation.

[0343]

[0344] Here, the parameter c(i) represents a weighting coefficient. Generally, c(i) is set so that c(i)≧0 and the sum of c(i) is 1. However, the method for setting c(i) is not limited to this.

[0345] Note that information I(n) indicating the properties of the direct sound is calculated every 5 ms that the direct sound is captured. In other words, it is possible to calculate the time variation of information I(n) indicating the properties of the direct sound with low delay. Therefore, this method is suitable for application requiring real-time performance.

[0346] Similarly to Equation 2, information I(n) indicating the properties of the direct sound may be calculated according to the following equation:

[0347]

[0348] Here, the parameter d(i) represents a weighting coefficient. Generally, d(i) is set so that d(i)≧0 and the sum of d(i) is 1. However, the method for setting d(i) is not limited to this.

[0349] In this equation, information I(n) indicating the properties of the direct sound is calculated recursively, making it possible to calculate the average energy over a long time period with a small amount of calculation.

[0350] The above equations 3 and 4 can be considered as filters in which E(n) is the input signal and I(n) is the output signal. In this case, equation 3 is a moving average (MA) model filter, and equation 4 is an autoregressive (AR) model filter, both of which have low-pass filter characteristics. Alternatively, an ARMA model filter, which is a combination of both, may be used.

[0351] In the above, a filter having low-pass characteristics is used to calculate the low-frequency components of the direct sound, but the method for calculating the low-frequency components of the direct sound is not limited to this. Furthermore, the method for deriving information indicating the time variation of the direct sound is not limited to the above-described formula or filter, and other known methods may be used. For example, the spectrum of the direct sound may be calculated by performing a frequency conversion on the direct sound. Then, the energy or average amplitude of the low-frequency components of the spectrum may be calculated.

[0352] In the above, the MA model or the AR model is used to derive the information indicating the time variation of the direct sound. The coefficients of these models may be predetermined fixed values ​​or may be variable values ​​that change over time.

[0353] The relationship between the analysis frame length and the interval at which the information update thread occurs may be as follows:

[0354] For example, if the time length of the analysis frame is TA (msec) and the occurrence interval of the information update thread is TU (msec), the value of N in the above (Equation 1) and (Equation 3) for the MA filter may be approximately the value given by TU / TA. Also, the values ​​of b(i) and d(i) (1≦i<N) in the above (Equation 2) and (Equation 4) for the AR filter may be values ​​such that the time constant of the filter is approximately TU (msec).

[0355] The reason for the above setting is that the filter is expected to converge within the interval period of information update.

[0356] On the other hand, if the value of the information indicating the time variation of the direct sound fluctuates too sharply with the above settings, I(n) may be calculated in advance. Then, the pre-calculated I(n) may be applied to the selection process of the reflected sounds. For example, I(t+tau) may be used in the processing of the t-th frame. Here, tau is a value determined according to the convergence characteristics of the filter. When convergence is slow, the value of tau is larger than when convergence is fast.

[0357] Furthermore, auditory masking (frequency masking) information calculated from the direct sound may be used as information indicating the characteristics of the direct sound. The auditory masking information indicates a threshold value for the amplitude value in the frequency domain that is masked by the direct sound. The amplitude value of the reflected sound in the same frequency domain may be compared with the threshold value, and processing may be performed to not select reflected sounds with amplitude values ​​smaller than the threshold value. The amplitude value of the reflected sound in the frequency domain may be acquired by the analysis unit 1301 as information indicating the characteristics of the reflected sound.

[0358] In this way, by setting the threshold value used to select reflected sounds according to the properties of the direct sound, it becomes possible to appropriately select reflected sounds that are auditorily necessary, and it becomes possible to effectively reflect the characteristics of hearing in the stereophonic sound reproduction system 1000. The process of detecting the properties of the direct sound, the process of determining the threshold value according to the properties, and the process of adjusting the threshold value according to the properties may be performed during the rendering process or before the rendering process starts.

[0359] For example, these processes may be performed when the virtual space is created (when the software is created), when processing of the virtual space starts (when the software is launched or rendering starts), or when an information update thread that occurs periodically in processing of the virtual space occurs, etc. Furthermore, when the virtual space is created may be when the virtual space is constructed before the start of acoustic processing, or when information about the virtual space (spatial information) is acquired, or when the software is acquired.

[0360] Here, in the information update thread, processing for updating the spatial information managed by the spatial information management units 1201 and 1211 is carried out.

[0361] The role of the information update thread is, for example, to update the position and orientation of the listener's avatar placed in the virtual space based on the position and orientation of the VR goggles worn by the listener, or to update the position of an object moving in the virtual space, etc. Such processing is handled within a processing thread that runs at a relatively low frequency of about several tens of Hz.

[0362] The process of updating information indicating the characteristics of the direct sound may be performed in such a processing thread that occurs less frequently. This is because the characteristics of the direct sound change less frequently than the frequency with which audio processing frames for audio output occur. This makes it possible to relatively reduce the computational load of this process. Furthermore, updating information at an unnecessarily high frequency poses a risk of generating pulsive noise. Updating information at a low frequency makes it possible to avoid such a risk.

[0363] [Third Modification of Threshold Setting Method] As another example of a method for setting a threshold, the threshold may be set according to the computational resources (CPU power, memory resources, PC performance, remaining battery power, etc.) used to process the reproduction of the virtual space. More specifically, the sensor 1405 of the audio signal processing device 1001 detects the amount of computational resources, and if the amount of computational resources is low, the threshold is set high. As a result, the volume of more reflected sounds becomes lower than the threshold, making it possible to reduce the amount of reflected sounds that are subjected to binaural processing, and thereby reducing the amount of computation.

[0364] Alternatively, when signal processing is performed in a device powered by a battery, such as a smartphone or VR goggles, it is expected that priority will be given to continuing processing for a long period of time and that computational resources will be saved. In such a case, the threshold may be set high without detecting the amount or remaining amount of computational resources.

[0365] [Fourth variant of threshold setting method] As another example of a threshold setting method, the audio signal processing device 1001 or the audio presentation device 1002 may be provided with a threshold setting unit (not shown), so that the threshold can be set by an administrator or listener of the virtual space.

[0366] For example, a listener wearing the audio presentation device 1002 may be able to select between an "energy saving mode" with less target reflected sounds and less computational effort, and a "high performance mode" with more target reflected sounds and more computational effort. Alternatively, the mode may be selectable by an administrator managing the stereophonic sound reproduction system 1000 or a creator of the stereophonic content. Alternatively, the threshold or threshold data may be directly selectable instead of the mode.

[0367] [First Modification of Operation of Rendering Unit] Fig. 20 is a flowchart showing a first modification of the operation of the audio signal processing device 1001. Fig. 20 shows the processing executed mainly by the rendering unit 1300 of the audio signal processing device 1001. In this modification, a volume compensation processing is added to the operation of the rendering unit 1300.

[0368] For example, the analysis unit 1301 acquires data (input signal) (S301). Next, the analysis unit 1301 analyzes the data (S302). Next, the selection unit 1302 determines whether or not to select reflected sound based on the analysis result (S303). Next, the playback unit 1303 performs volume compensation processing based on the reflected sound that is not selected (S304). Next, the playback unit 1303 performs acoustic processing on the direct sound and reflected sound (S305). Then, the playback unit 1303 outputs the direct sound and reflected sound as audio (S306).

[0369] Of the above processes (S301 to S306), the processes other than the volume compensation process (S304) are common to the other examples described above, and therefore description thereof will be omitted.

[0370] The volume compensation process is performed in response to reflected sounds that were not selected in the selection process. For example, a lack of perceived loudness occurs when reflected sounds are not selected in the selection process. The volume compensation process suppresses the sense of discomfort that accompanies such a lack of perceived loudness. The following two methods are disclosed as examples of methods for compensating for perceived loudness. Either of the two methods may be used.

[0371] First, a method for compensating for the sense of volume by increasing the volume of the direct sound will be described. The reproduction unit 1303 generates a direct sound by increasing the volume of the direct sound by the amount of the volume of the unselected reflected sound. This compensates for the sense of volume that would be lost by not generating reflected sound.

[0372] When increasing the volume, the playback unit 1303 may increase the volume for each frequency component in accordance with the frequency characteristics of the reflected sound. To enable such processing, a volume attenuation rate at which the reflective object attenuates the volume may be assigned to each predetermined frequency band. This makes it possible to derive the frequency characteristics of the reflected sound.

[0373] Next, a method for compensating for the perceived loudness by synthesizing reflected sounds with direct sounds will be described. In this method, the playback unit 1303 adds unselected reflected sounds to the direct sound to generate a direct sound, thereby compensating for the perceived loudness caused by not generating reflected sounds. The generated direct sound reflects the volume (amplitude), frequency, delay, etc. of the unselected reflected sounds.

[0374] In the case of the method of increasing the volume of direct sound, the amount of calculation required for the compensation process is extremely small, but only the volume is compensated. In the case of the method of combining direct sound with reflected sound, the amount of calculation required for the compensation process is greater than in the method of increasing the volume of direct sound, but the characteristics of the reflected sound are compensated more accurately.

[0375] In either case, the overall amount of calculation is reduced because only direct sound is generated, without generating reflected sound. In particular, the amount of calculation required for binaural processing, including the process of convolving HRTFs, is reduced, resulting in a significant reduction in the overall amount of calculation. This is because the amount of calculation required for binaural processing is far greater than the amount of calculation required for the compensation process described above.

[0376] If the reason why the reflected sound is not selected is that the volume of the reflected sound is below the masking threshold, the perceived volume is not lost, so the reflected sound may simply be removed without performing compensation processing.

[0377] [Second Modification of Operation of Rendering Unit] Fig. 21 is a flowchart showing a second modification of the operation of the audio signal processing device 1001. Fig. 21 shows the processing executed mainly by the rendering unit 1300 of the audio signal processing device 1001. In this modification, left-right volume difference adjustment processing is added to the operation of the rendering unit 1300.

[0378] For example, the analysis unit 1301 analyzes an input signal (S401). Next, the analysis unit 1301 detects the direction from which the sound is coming (S402). Next, the selection unit 1302 adjusts the difference in volume between the sounds perceived by the left and right ears (S403). The selection unit 1302 also adjusts the difference in arrival time (delay) between the sounds perceived by the left and right ears (S404). The selection unit 1302 determines whether to select a reflected sound based on the adjusted sound information (S405).

[0379] Of the above processes (S401 to S405), the processes other than the left-right volume difference adjustment process (S403) and the delay adjustment process (S404) are common to the other examples described above, and therefore descriptions thereof will be omitted.

[0380] Fig. 22 is a diagram showing an example of the arrangement of an avatar, a sound source object, and an obstacle object. For example, when the front direction of the listener is 0 degrees, and the polarity (e.g., positive or negative) of the incoming direction of the direct sound (θ) and the incoming direction of the reflected sound (γ) (direction of the reflected sound (γ)) is different, as shown in Fig. 22, the volume difference occurring between the two ears is corrected.

[0381] Specifically, when the polarities of θ and γ are different, the ear that primarily (first) perceives the direct sound and the reflected sound is different. In this case, the selection unit 1302 performs the left-right volume difference adjustment process (S403) to adjust the volume of the direct sound according to the position of the ear that primarily perceives the reflected sound. For example, the selection unit 1302 attenuates the volume of the direct sound when it reaches the listener by multiplying the volume by (1.0-0.3 sin(θ)) (0≦θ≦180).

[0382] The selection unit 1302 calculates the volume ratio between the volume of the direct sound corrected as described above and the volume of the reflected sound, and compares the calculated volume ratio with a threshold value to determine whether to select the reflected sound. This corrects the volume difference that occurs between the two ears, more accurately derives the volume of the direct sound that affects the reflected sound, and more accurately determines whether to select the reflected sound.

[0383] Furthermore, in addition to the left-right volume difference adjustment process (S403), the selection unit 1302 may also perform a delay adjustment process (S404) in which the selection unit 1302 delays the direct sound arrival time in accordance with the position of the ear that perceives the reflected sound. Specifically, the selection unit 1302 may delay the direct sound arrival time by adding (a(sin θ+θ) / c) ms (where a is the radius of the head and c is the speed of sound) to the direct sound arrival time.

[0384] [Third Modification of the Operation of the Rendering Unit] A method of setting a threshold value according to the direction of arrival will be described.

[0385] Fig. 23 is a flowchart showing yet another example of the selection process. A description of the process common to the example of Fig. 14 will be omitted. In the example of Fig. 23, the selection unit 1302 selects reflected sounds using a threshold value according to the arrival direction.

[0386] Specifically, the selection unit 1302 calculates the direct sound arrival direction (θ) and the reflected sound arrival direction (γ) (the direction of the reflected sound (γ)) determined using the avatar orientation as a reference, from the direct sound arrival path (pd) and the reflected sound arrival path (pr) calculated by the analysis unit 1301, and the avatar orientation information D1. That is, the selection unit 1302 detects the direct sound arrival direction (θ) and the reflected sound arrival direction (γ) (S231). The orientation of the avatar corresponds to the orientation of the listener. The avatar orientation information D1 may be included in the input signal.

[0387] The selection unit 1302 uses three indexes including the direct sound arrival direction (θ), the reflected sound arrival direction (γ), and the time difference (T) to identify the threshold to be used in the selection process from a three-dimensional array such as that shown in Figure 15 (S232).

[0388] As an example, a method for setting a threshold value used in the selection process when an avatar, a sound source object, and an obstacle object are arranged as shown in FIG. 22 will be described.

[0389] From the input signal, position information of the avatar, sound source object, and obstacle object, as well as avatar orientation information D1, are acquired. Using this position information and orientation information D1, the direction of the direct sound (θ) and the direction of the sound image of the reflected sound (γ) are calculated when the orientation of the avatar is set to 0 degrees. In the case of Figure 22, the direction of the direct sound (θ) is approximately 20 degrees, and the direction of the sound image of the reflected sound (γ) is approximately 265 degrees (-95 degrees).

[0390] 15, threshold values ​​are identified from an array region corresponding to the values ​​of the two directions (θ) and (γ) and the value of the time difference (T) calculated by the analysis unit 1301. If there is no index corresponding to the calculated values ​​of (θ), (γ), and (T), a threshold value corresponding to the closest index may be identified.

[0391] Alternatively, the threshold value may be determined by performing a process such as interpolation, extrapolation, or the like based on one or more threshold values ​​corresponding to one or more indexes close to the calculated values ​​of (θ), (γ), and (T). For example, a threshold value corresponding to (20°, 265°, T) may be determined based on four threshold values ​​corresponding to four indexes, namely, (0°, 225°, T), (0°, 270°, T), (45°, 225°, T), and (45°, 270°, T).

[0392] The selection process based on the difference between the angle (θ) of the direction from which the direct sound arrives and the angle (γ) of the direction from which the reflected sound arrives will be described.

[0393] For example, threshold data having the angular difference (Φ) between the arrival direction (θ) of the direct sound and the arrival direction (γ) of the reflected sound and the time difference (T) as a two-dimensional index array may be created and set in advance, as shown in Fig. 16. In this case, the angular difference (Φ) and the time difference (T) are referenced in the selection process. Alternatively, the angular difference (Φ) between the angle (θ) of the arrival direction of the direct sound and the angle (γ) of the arrival direction of the reflected sound may be calculated in the selection process, and the calculated angular difference (Φ) may be used to specify the threshold.

[0394] Alternatively, threshold data may be set that has, as an index array, a combination of the angle difference (Φ), the direction of arrival of the direct sound (θ), and the time difference (T), or a combination of the angle difference (Φ), the direction of arrival of the reflected sound (γ), and the time difference (T).

[0395] Alternatively, threshold data having the values ​​of (θ), (γ) and (T) as a three-dimensional index array as shown in FIG. 15 may be set.

[0396] [Fourth Modification of Operation of Rendering Unit] The processes performed by the analysis unit 1301, selection unit 1302, and reproduction unit 1303 described above may be performed as pipeline processes as described in, for example, Patent Document 3.

[0397] FIG. 24 is a block diagram showing an example of the configuration for the rendering unit 1300 to perform pipeline processing.

[0398] The rendering unit 1300 in Fig. 24 includes a reverberation processing unit 1311, an early reflection processing unit 1312, a distance attenuation processing unit 1313, a selection unit 1314, a generation unit 1315, and a binaural processing unit 1316. These multiple components may be configured from multiple components of the rendering unit 1300 shown in Fig. 7, or may be configured from at least some of multiple components of the audio signal processing device 1001 shown in Fig. 5.

[0399] Pipeline processing refers to dividing the process for applying sound effects into multiple processes and executing the multiple processes one by one in sequence. Each of the multiple processes performs, for example, signal processing on an audio signal or generation of parameters used in the signal processing.

[0400] The rendering unit 1300 may perform reverberation processing, early reflection processing, distance attenuation processing, binaural processing, and the like as pipeline processing. However, these processes are merely examples, and the pipeline processing may include other processes or may not include some of the processes. For example, the pipeline processing may include diffraction processing and occlusion processing. Furthermore, for example, reverberation processing may be omitted if it is not necessary.

[0401] Each process may be expressed as a stage. An audio signal such as a reflected sound generated as a result of each process may be expressed as a rendering item. The multiple stages in the pipeline process and their order are not limited to the example shown in FIG. 24 .

[0402] Here, the parameters used in the selection process (arrival paths, arrival times, and volume ratios for direct sound and reflected sound) are calculated in one of multiple stages for generating a rendering item. In other words, the parameters used to select reflected sounds are calculated as part of the pipeline processing for generating a rendering item. Note that not all stages need to be performed by the rendering unit 1300. For example, some stages may be omitted or may be performed by a unit other than the rendering unit 1300.

[0403] The following describes reverberation processing, early reflection processing, distance attenuation processing, selection processing, generation processing, and binaural processing that may be included as stages in the pipeline processing. At each stage, metadata included in the input signal may be analyzed to calculate parameters used to generate reflected sounds.

[0404] In the reverberation processing, the reverberation processor 1311 generates an audio signal indicating a reverberant sound or parameters used to generate an audio signal. A reverberant sound is a sound that arrives at a listener as reverberation after a direct sound. As an example, a reverberant sound is a sound that arrives at a listener after a relatively late stage (e.g., about 150 ms after the arrival of the direct sound) after an early reflected sound (described later) arrives at the listener, and after having been reflected more times (e.g., several tens of times) than an early reflected sound.

[0405] The reverberation processor 1311 refers to the audio signal and spatial information contained in the input signal, and calculates the reverberation sound using a predetermined function prepared in advance as a function for generating the reverberation sound.

[0406] The reverberation processor 1311 may generate reverberant sounds by applying a known reverberation generation method to the audio signal included in the input signal. An example of a known reverberation generation method is the Schroeder method, but known reverberation generation methods are not limited to the Schroeder method. Furthermore, when applying a known reverberation generation method, the reverberation processor 1311 uses the shape and acoustic characteristics of the sound reproduction space indicated by the spatial information. This allows the reverberation processor 1311 to calculate parameters for generating reverberant sounds.

[0407] In the early reflection process, the early reflection processor 1312 calculates parameters for generating early reflection sounds based on spatial information. The early reflection sounds are reflected sounds that arrive at the listener after one or more reflections at a relatively early stage after a direct sound from a sound source object arrives at the listener (for example, about several tens of milliseconds after the direct sound arrives).

[0408] The early reflection processing unit 1312 refers to, for example, the audio signal and metadata, and calculates the path of the reflected sound that travels from the sound source object to the listener after being reflected by the reflecting object. For example, the path calculation may use the shape of the three-dimensional sound field (space), the size of the three-dimensional sound field, the positions of reflecting objects such as structures, and the reflectance of the reflecting object.

[0409] The early reflection processing unit 1312 may also calculate the path of the direct sound. Information about the path may be used as a parameter by which the early reflection processing unit 1312 generates the early reflected sound, or may be used as a parameter by which the selection unit 1314 selects the reflected sound.

[0410] In the distance attenuation process, the distance attenuation processor 1313 calculates the volume of the direct sound and the reflected sound that reach the listener based on the path lengths of the direct sound and the reflected sound. The volume of the direct sound and the reflected sound that reach the listener attenuates in proportion to the distance of the path to the listener (inversely proportional to the distance) relative to the volume of the sound source. Therefore, the distance attenuation processor 1313 can calculate the volume of the direct sound by dividing the volume of the sound source by the path length of the direct sound, and can calculate the volume of the reflected sound by dividing the volume of the sound source by the path length of the reflected sound.

[0411] In the selection process, the selection unit 1314 selects a generation target reflected sound based on parameters calculated before the selection process. Any of the selection methods disclosed herein may be used to select the generation target reflected sound.

[0412] The selection process may be performed on all reflected sounds, or may be performed only on reflected sounds with high evaluation values ​​based on the evaluation process as described above. In other words, reflected sounds with low evaluation values ​​may be determined not to be selected without even undergoing the selection process. For example, a reflected sound with a very low volume may be considered to have a low evaluation value and may be determined not to be selected.

[0413] Alternatively, for example, a selection process may be performed on all reflected sounds, and the evaluation values ​​of the reflected sounds selected in the selection process may be determined, and reflected sounds with low evaluation values ​​may be re-determined as not being selected.

[0414] The selection process and the evaluation process may be performed independently or in combination. When the selection process and the evaluation process are performed in combination, either of the two processes may be performed first.

[0415] In the generation process, the generation unit 1315 generates direct sound and reflected sound. For example, the generation unit 1315 generates direct sound from an audio signal included in the input signal based on the arrival time and volume of the direct sound at the time of arrival. Furthermore, for the reflected sound selected in the selection process, the generation unit 1315 generates reflected sound from an audio signal included in the input signal based on the arrival time and volume of the reflected sound at the time of arrival.

[0416] In the binaural processing, the binaural processing unit 1316 performs signal processing so that the audio signal of the direct sound is perceived by the listener as a sound arriving from the direction of the sound source object. Furthermore, the binaural processing unit 1316 performs signal processing so that the reflected sound selected by the selection unit 1314 is perceived by the listener as a sound arriving from the reflecting object.

[0417] For example, the binaural processing unit 1316 performs processing to apply the HRIR DB based on the position and orientation of the listener in the sound space so that sound arrives at the listener from the position of a sound source object or the position of an obstacle object.

[0418] HRIR (Head-Related Impulse Responses) is a response characteristic when one impulse is generated. Specifically, HRIR is a response characteristic obtained by converting a head-related transfer function, which represents changes in sound caused by surrounding objects including the auricle, the human head, and shoulders, from a frequency domain representation to a time domain representation by Fourier transform. The HRIR DB is a database containing such information.

[0419] Furthermore, the position and orientation of the listener in the sound space are, for example, the position and orientation of the virtual listener in the virtual sound space. The position and orientation of the virtual listener in the virtual sound space may change in accordance with the movement of the listener's head. The position and orientation of the virtual listener in the virtual sound space may also be determined based on information acquired from the sensor 1405.

[0420] The programs, spatial information, HRIR DB, threshold data, and other parameters used in the above processing are obtained from the memory 1404 provided in the audio signal processing device 1001 or from outside the audio signal processing device 1001.

[0421] The pipeline processing may also include other processes. The rendering unit 1300 may also include processing units (not shown) for performing other processes included in the pipeline processing. For example, the rendering unit 1300 may include a diffraction processing unit and an occlusion processing unit.

[0422] The diffraction processing unit executes processing to generate an audio signal representing a sound including diffracted sound caused by an obstacle object between the listener and the sound source object in a three-dimensional sound field (space). When an obstacle object exists between the sound source object and the listener, the diffracted sound is a sound that travels from the sound source object to the listener, going around the obstacle object.

[0423] The diffraction processing unit calculates a path of the diffracted sound from the sound source object to the listener, bypassing the obstacle object, and generates the diffracted sound based on the path, for example, by referring to the audio signal and metadata. The path calculation may use the positions of the sound source object, the listener, and the obstacle object in the three-dimensional sound field (space), as well as the shape and size of the obstacle object.

[0424] When a sound source object is present on the other side of an obstacle object, the occlusion processing unit generates an audio signal of the sound that leaks from the sound source object and passes through the obstacle object based on spatial information and information such as the material of the obstacle object.

[0425] [Example of Sound Source Object] In the above, the position information assigned to the sound source object indicates a "point" in the virtual space as the position of the sound source object. That is, in the above, the sound source is defined as a "point sound source."

[0426] On the other hand, a sound source in a virtual space may be defined as an object having length, size, shape, etc., i.e., as a spatially extended sound source rather than a point sound source. In this case, the distance between the listener and the sound source and the direction from which the sound is coming are not determined. Therefore, reflected sounds caused by such sound sources may be limited to those selected by the selection unit 1302 without being analyzed by the analysis unit 1301 or regardless of the analysis results. This makes it possible to avoid deterioration in sound quality that may occur when reflected sounds are not selected.

[0427] Alternatively, a representative point such as the center of gravity of the object may be determined, and the processing of the present disclosure may be applied on the assumption that the sound is generated from that representative point. In this case, the threshold may be adjusted according to information on the spatial extent of the sound source.

[0428] [Examples of Direct Sound and Reflected Sound] For example, direct sound is sound that is not reflected by a reflecting object, and reflected sound is sound that is reflected by a reflecting object. Direct sound may be sound that arrives at the listener from a sound source without being reflected by a reflecting object, or reflected sound may be sound that arrives at the listener from a sound source after being reflected by a reflecting object.

[0429] Furthermore, the direct sound and the reflected sound are not limited to sounds that have arrived at the listener, but may be sounds that have not yet arrived at the listener. For example, the direct sound may be sounds output from a sound source, or in other words, sounds from the sound source.

[0430] 25 is a diagram illustrating sound transmission and diffraction. As shown in FIG. 25, there are cases where direct sound does not reach the listener due to the presence of an obstacle object between the sound source object and the listener. In this case, sound emitted from the sound source object, transmitted through the obstacle object, and reached the listener may be considered as direct sound. Meanwhile, sound emitted from the sound source object, diffracted by the obstacle object, and reached the listener may be considered as reflected sound.

[0431] Furthermore, the two sounds compared in the selection process are not limited to a direct sound and a reflected sound based on a sound emitted from a single sound source. For example, a sound may be selected by comparing two reflected sounds based on a sound emitted from a single sound source. In this case, the direct sound in the present disclosure may be interpreted as the sound that reaches the listener first, and the reflected sound in the present disclosure may be interpreted as the sound that reaches the listener later.

[0432] [Example of Bitstream Structure] A bitstream includes, for example, an audio signal and metadata. The audio signal is sound data that expresses sound, and indicates information about the frequency and intensity of the sound. The metadata includes spatial information about the sound space, which is the space of the sound field.

[0433] For example, the spatial information is information about a space in which a listener who listens to a sound based on an audio signal is located. Specifically, the spatial information is information about a predetermined position (localization position) for localizing a sound image at a predetermined position in a sound space (e.g., a three-dimensional sound field), that is, for allowing the listener to perceive a sound arriving from a direction corresponding to the predetermined position. The spatial information includes, for example, sound source object information and position information indicating the position of the listener.

[0434] The sound source object information is information about a sound source object that generates a sound based on an audio signal. That is, the sound source object information is information about an object (sound source object) that reproduces an audio signal, and is information about a virtual sound source object that is placed in a virtual sound space. Here, the virtual sound space may correspond to a real space in which an object that generates a sound is placed, and the sound source object in the virtual sound space may correspond to an object that generates a sound in the real space.

[0435] The sound source object information may indicate the position of the sound source object arranged in the sound space, the orientation of the sound source object, the directivity of the sound emitted by the sound source object, whether the sound source object belongs to a living thing or not, whether the sound source object is a moving object or not, etc. For example, the audio signal is associated with one or more sound source objects indicated by the sound source object information.

[0436] The bitstream has a data structure that is made up of, for example, metadata (control information) and an audio signal.

[0437] The audio signal and metadata may be contained in a single bitstream or in separate bitstreams, or may be contained in a single file or in separate files.

[0438] A bitstream may exist for each sound source or for each playback time. Even if a bitstream exists for each playback time, multiple bitstreams may be processed in parallel at the same time.

[0439] Metadata may be assigned to each bitstream, or may be assigned to multiple bitstreams together as information for controlling multiple bitstreams. In this case, multiple bitstreams may share the same metadata. Metadata may also be assigned for each playback time.

[0440] When multiple bitstreams or multiple files exist, one or more of the bitstreams or files may contain information indicating the associated bitstreams or files, or alternatively, each of all of the bitstreams or each of all of the files may contain information indicating the associated bitstreams or files.

[0441] Here, the related bitstreams or related files are, for example, bitstreams or files that may be used simultaneously during audio processing, and may also include bitstreams or files that collectively describe information indicating related bitstreams or related files.

[0442] Here, the information indicating the related bitstream or related file may be, for example, an identifier indicating the related bitstream or related file. Alternatively, the information indicating the related bitstream or related file may be, for example, a file name indicating the related bitstream or related file, a URL (Uniform Resource Locator), or a URI (Uniform Resource Identifier).

[0443] In this case, the acquisition unit identifies and acquires the related bitstream or related file based on the information indicating the related bitstream or related file. Alternatively, the bitstream or file may contain information indicating the related bitstream or related file, and another bitstream or another file may contain information indicating the related bitstream or related file.

[0444] Here, the file containing information indicating the associated bitstream or associated file may be a control file such as a manifest file used for content distribution.

[0445] Note that all or part of the metadata may be obtained from sources other than the bitstream of the audio signal. For example, either the metadata for controlling the sound or the metadata for controlling the video may be obtained from sources other than the bitstream, or both may be obtained from sources other than the bitstream.

[0446] Furthermore, metadata for controlling the video may be included in the bitstream acquired by the stereophonic sound reproduction system 1000. In this case, the stereophonic sound reproduction system 1000 may output the metadata for controlling the video to a display device that displays images or a stereophonic video reproduction device that reproduces the stereophonic video.

[0447] [Examples of Information Included in Metadata] Metadata may be information used to describe a scene represented in a sound space, where a scene is a term that refers to a collection of all elements representing three-dimensional video and sound events in a sound space that is modeled by the stereophonic sound reproduction system 1000 using the metadata.

[0448] That is, the metadata may include not only information for controlling audio processing but also information for controlling video processing. The metadata may include only one of information for controlling audio processing and information for controlling video processing, or may include both.

[0449] The stereophonic sound reproduction system 1000 generates virtual sound effects by performing sound processing on audio signals using metadata included in the bitstream and additionally acquired interactive listener position information, etc. Among the sound effects, early reflection processing, obstacle processing, diffraction processing, blocking processing, and reverberation processing may be performed, and other sound processing may be performed using the metadata. For example, sound effects such as distance attenuation, localization, or Doppler effect may be added.

[0450] Furthermore, information on switching on / off all or part of the sound effects, or priority information for multiple sound effect processes may be added to the metadata.

[0451] As an example, the metadata includes information about a sound space including sound source objects and obstacle objects, and information about a positioning position for localizing a sound image at a predetermined position within the sound space (i.e., allowing the listener to perceive sound coming from a predetermined direction).

[0452] Here, an obstacle object is an object that may affect the sound perceived by the listener by, for example, blocking or reflecting the sound emitted by the sound source object before it reaches the listener. Obstacle objects may include not only stationary objects but also moving objects such as animals or machines. The animal may also be a person, etc.

[0453] Furthermore, when multiple sound source objects exist in a sound space, other sound source objects can be obstacle objects for any sound source object. In other words, both non-sound-emitting objects, such as building materials or inanimate objects, which do not emit sound, and sound source objects that emit sound can be obstacle objects.

[0454] The metadata includes information that represents all or part of the shape of the sound space, the shape and position of obstacle objects in the sound space, the shape and position of sound source objects in the sound space, and the position and orientation of the listener in the sound space.

[0455] The sound space may be either a closed space or an open space. The metadata may also include information indicating the reflectance of obstacle objects that may reflect sound in the sound space. For example, the floor, walls, or ceiling that form the boundaries of the sound space may also constitute obstacle objects.

[0456] The reflectance is the ratio of the energy of reflected sound to incident sound, and may be set for each frequency band of sound. Of course, the reflectance may be set uniformly regardless of the frequency band of sound. Note that when the sound space is an open space, parameters such as a uniform attenuation rate, diffracted sound, and early reflected sound may be used.

[0457] The metadata may include information other than reflectance as a parameter related to an obstacle object or a sound source object. For example, the metadata may include information related to the material of the object as a parameter related to both a sound source object and a non-sound-producing object. Specifically, the metadata may include information such as diffusion rate, transmittance, and sound absorption rate.

[0458] The information about the sound source object may include information indicating the volume, radiation characteristics (directivity), playback conditions, the number and type of sound sources in one object, and the sound source area in the object. The playback conditions may determine, for example, whether the sound is a continuous sound or an event-triggering sound. The sound source area in the object may be determined based on the relative relationship between the position of the listener and the position of the object, or may be determined using the object as a reference.

[0459] For example, if the sound source area is defined relative to the listener's position and the object's position, it is possible for the listener to perceive sound E coming from the right side of the object and sound F coming from the left side of the object.

[0460] Furthermore, when a sound source region is defined using an object as a reference, it is possible to fix which region of the object will emit which sound. For example, when a listener views an object from the front, it is possible for the listener to perceive a high-pitched sound from the right side of the object and a low-pitched sound from the left side of the object. When a listener views an object from the back, it is possible for the listener to perceive a low-pitched sound from the right side of the object and a high-pitched sound from the left side of the object.

[0461] The spatial metadata may include the time to early reflections, the reverberation time, the ratio of direct sound to diffuse sound, etc. If the ratio of direct sound to diffuse sound is zero, the listener will perceive only direct sound.

[0462] [Brief Summary] Here, the present embodiment will be briefly summarized.

[0463] The relationship between direct sound and reflected sound is analyzed, and when the direct sound is considered to be the preceding sound and the reflected sound is considered to be the following sound, if the relationship is such that the precedence effect occurs, in other words, when the reflected sound is below the echo detection limit, the reflected sound will not be perceived, and therefore even if the reflected sound is deleted, there will be little impact on the listener's hearing.

[0464] Fig. 26 is a diagram showing an example of the positional relationship between a listener and an obstacle object according to this embodiment. Fig. 27 is a diagram showing another example of the positional relationship between a listener and an obstacle object according to this embodiment. Note that Fig. 26 is equivalent to the positional relationship shown in Fig. 9, and Fig. 27 is equivalent to the positional relationship shown in Fig. 10. Fig. 28 is an example of an echo detection limit threshold according to this embodiment. Note that the echo detection limit threshold shown in Fig. 28 is an example of the threshold data shown in Fig. 12C etc.

[0465] For example, when comparing the positional relationship in Fig. 26 with the positional relationship in Fig. 27, the volume of the reflected sound heard by the listener in the positional relationship in Fig. 26 is lower than the volume of the reflected sound heard by the listener in the positional relationship in Fig. 27. This is because the path length over which the reflected sound arrives in the positional relationship in Fig. 26 is longer than the path length over which the reflected sound arrives in the positional relationship in Fig. 27.

[0466] Therefore, when judging only by the volume of the reflected sound, the reflected sound shown in Fig. 26 has a smaller auditory impact than the reflected sound shown in Fig. 27. However, when comparing the arrival time of the reflected sound at the listening position shown in Fig. 26 with the arrival time of the reflected sound at the listening position shown in Fig. 27, the case shown in Fig. 26 is slower.

[0467] For this reason, when a judgment is made from the viewpoint of the echo detection limit, as shown in FIG. 28, the reflected sound shown in FIG. 27 is below the echo detection limit and is therefore not perceived by the listener as a reflected sound, whereas the reflected sound shown in FIG. 26 is above the echo detection limit and is therefore perceived by the listener as a reflected sound.

[0468] In this embodiment, this is utilized to determine the auditory importance of reflected sounds and prevent unimportant reflected sounds from being reproduced, thereby reducing the amount of calculation required for processing reflected sounds.

[0469] The present embodiment can be briefly summarized as above.

[0470] Further consideration can be given as follows.

[0471] As described above, when a direct sound is a preceding sound and a reflected sound is a following sound, if the relationship is such that the precedence effect occurs, the sound image position of the reflected sound is moved to the position of the direct sound, and the audio signal representing the reflected sound and the audio signal representing the direct sound are synthesized (merged) into a single audio signal, thereby reducing the amount of processing related to the reflected sound. Furthermore, in such cases, even if a single merged audio signal is used, the listener is unlikely to feel uncomfortable.

[0472] Here, the arrival time difference between direct sound and reflected sound is considered. The arrival time difference is shorter in the positional relationship shown in Fig. 27 than in the positional relationship shown in Fig. 26. For example, consider a case where the precedence effect does not occur in the case shown in Fig. 26, but occurs in the case shown in Fig. 27. In this case, in the case shown in Fig. 27, the listener feels as if the sound image position of the reflected sound has moved to the sound image position of the direct sound.

[0473] By utilizing this, when the conditions for the precedence effect are met, the direct sound and the reflected sound are combined, i.e., a single audio signal (merge audio signal) representing a single synthesized sound (merge sound) is generated, thereby reducing the amount of calculation required for processing the reflected sound. In this case, the volume of the synthesized sound is the sum of the volume of the direct sound and the volume of the reflected sound.

[0474] Fig. 29 is a diagram showing an example in which the sound image position of the reflected sound is moved in the positional relationship shown in Fig. 27. When the precedence effect occurs under the conditions shown in Fig. 27, as shown in Fig. 29, the sound image position of the reflected sound is moved to the sound image position of the direct sound, and the volume of the direct sound is increased by the volume of the reflected sound. As a result, the processing related to the reflected sound can be reduced.

[0475] However, the process of generating a merge audio signal only when a precedence effect occurs may cause the following problems, which will be explained with reference to FIGS.

[0476] 30 is a diagram showing an example in which direct sound and indirect sound arrive at the listener from the same direction. In FIG. 30, the indirect sound is a reflected sound reflected by a wall.

[0477] 31 is a diagram showing an example in which direct sound and indirect sound arrive at a listener from one sound source (sound source 101) and another sound source (sound source 102). The one sound source (sound source 101) and the other sound source (sound source 102) emit different types of sounds. For example, the one sound source (sound source 101) emits a sound representing a dog barking, and the other sound source (sound source 102) emits the sound of a running vehicle. As shown in FIG. 31, the indirect sound (reflected sound) from the one sound source (sound source 101) and the direct sound from the other sound source (sound source 102) arrive at the listener from the same direction.

[0478] 32 is a diagram showing an example in which transmitted sound and diffracted sound, which are sounds emitted from one sound source, arrive at a listener. In FIG. 32, the transmitted sound is the preceding sound, and the diffracted sound, which is an indirect sound, is the following sound.

[0479] 30 to 32, the direct sound and the indirect sound arrive at the listener from the same direction or approximately the same direction. In such cases, it should be possible to combine the direct sound and the indirect sound, that is, to perform processing to merge the audio signal representing the indirect sound and the audio signal representing the direct sound into a single audio signal. This is because performing this processing rarely causes discomfort to the listener and can reduce processing related to reflected sound.

[0480] However, there are cases where it is not possible to determine whether or not merging processing should be performed based solely on the precedence effect condition.

[0481] An example of a case where a determination cannot be made is when the arrival time difference between the direct sound and the indirect sound (reflected sound) is too short or too long, which does not meet the conditions for the precedence effect, as shown in the example of Fig. 30. Another example of a case where a determination cannot be made is when the indirect sound and the direct sound are not similar, that is, when the indirect sound is not derived from the direct sound, as shown in the example of Fig. 31. Another example of a case where a determination cannot be made is when the volume of the preceding sound is lower than the volume of the following sound, as shown in the example of Fig. 32.

[0482] In these cases, the precedence effect does not occur, and therefore the above-described process of generating a merge audio signal only when the precedence effect occurs cannot generate a merge audio signal, meaning that the processing related to reflected sounds cannot be reduced. In other words, such an audio signal processing method poses the problem of being unable to reduce the amount of calculation and the calculation load.

[0483] Therefore, in the following, a more detailed description will be given of an audio signal processing method that can reduce the amount of calculation and the calculation load in the sound space.

[0484] (Embodiment 2) The following describes embodiment 2. The following mainly describes the differences from embodiment 1, and the description of commonalities will be omitted or simplified.

[0485] [Configuration of the Rendering Unit] First, a description will be given of the configuration of the rendering unit 2300 according to this embodiment. Fig. 33 is a block diagram showing an example of the configuration of the rendering unit 2300 according to this embodiment.

[0486] The rendering unit 2300 includes an analysis unit 2301, a selection unit 2302, and a reproduction unit 2303. The audio signal processing device according to this embodiment is an example of a decoding device, and the decoding device includes a decoder, which includes the rendering unit 2300. In other words, it can be said that the audio signal processing device according to this embodiment includes the analysis unit 2301, the selection unit 2302, and the reproduction unit 2303. The rendering unit 2300 applies acoustic processing to sound data included in an input signal and outputs the result.

[0487] As in the first embodiment, the input signal is composed of, for example, spatial information, sensor information, and sound data. The spatial information also includes physical information such as the reflection coefficient, transmission coefficient, and diffraction coefficient of non-sound-producing objects (obstacle objects).

[0488] In this embodiment, the description will be mainly given using reflected sound, which is an example of indirect sound, but the same processing is performed even if indirect sound is used instead of reflected sound. In addition, examples of indirect sound include reflected sound and diffracted sound.

[0489] The analysis unit 2301 may be able to perform all or part of the processing performed by the analysis unit 1301 according to embodiment 1. Similarly to the analysis unit 1301 according to embodiment 1, the analysis unit 2301 analyzes the audio signal included in the input signal and the spatial information received from the spatial information management units 1201 and 1211. As a result, the analysis unit 2301 calculates information necessary for generating direct sound and reflected sound in the reproduction unit 2303, as well as information necessary for selecting whether or not to generate reflected sound. The method by which the analysis unit 2301 calculates this information is as described in embodiment 1.

[0490] The analysis unit 2301 also performs analysis processing of the input signal, as performed by the analysis unit 1301 according to the first embodiment in S101 of Fig. 8. That is, the analysis unit 2301 analyzes the input signal input to the audio signal processing device according to the present embodiment, and detects direct sound and reflected sound that may occur in the sound space.

[0491] In the present embodiment, for example, an indirect sound (more specifically, a reflected sound) is an example of a first sound, and a direct sound is an example of a second sound. The first sound and the second sound are different sounds.

[0492] When a direct sound and a reflected sound are detected, the analysis unit 2301 generates an audio signal indicating the reflected sound and an audio signal indicating the direct sound based on the spatial information and the sound data.

[0493] More specifically, the analysis unit 2301 creates an audio signal indicating reflected sound and an audio signal indicating direct sound based on the position information of the sound source object contained in the spatial information, the position information of the non-sound-emitting object (obstacle object), the position information and physical information of the listener, and the sound data.

[0494] That is, the analysis unit 2301 generates an audio signal to be generated in the virtual space based on the spatial information and the sound data. Here, an audio signal representing a first sound (first audio signal) and an audio signal representing a second sound (second audio signal) are generated. In this embodiment, the first audio signal corresponds to an audio signal representing a reflected sound, and the second audio signal corresponds to an audio signal representing a direct sound.

[0495] Furthermore, the audio signal created by the analysis unit 2301 is provided with attribute information that identifies the attribute of the audio signal, i.e., the analysis unit 2301 creates an audio signal that includes such attribute information. The analysis unit 2301 also creates attribute information. An audio signal that includes attribute information is created for each sound generated in the virtual space.

[0496] As described above, the first sound (reflected sound (indirect sound)) or the second sound (direct sound) is indicated by an audio signal, but is not limited to this, and the attributes of the audio signal may include information indicating whether the audio indicated by the audio signal is the first sound (reflected sound (indirect sound)) or the second sound (direct sound).

[0497] An audio signal whose attribute includes information indicating reflected sound (indirect sound) may be described as an audio signal indicating reflected sound (indirect sound), and an audio signal whose attribute includes information indicating direct sound may be described as an audio signal indicating direct sound.

[0498] The first audio signal is an audio signal including first attribute information. The first attribute information is information specifying an attribute of the first audio signal. The attribute may include information indicating a first sound. The second audio signal is an audio signal including second attribute information. The second attribute information is information specifying an attribute of the second audio signal. The attribute may include information indicating a second sound.

[0499] Furthermore, the attribute information may include information necessary for radiating an audio signal into a sound space, such as gain information, information on gain characteristics for each frequency bandwidth, position information, and directivity information. That is, the necessary information may be held in the attribute information. Furthermore, the attribute information may be linked to the audio signal as metadata. The gain characteristics for each frequency bandwidth of the audio signal included in the attribute information may be identified, for example, based on spatial information included in the input information. Information indicating frequency characteristics that indicate auditory sensitivity may be identified, for example, based on spatial information included in the input information, and may particularly be identified as information linked to the listener's avatar.

[0500] A sound that reaches the listener's head directly from a single sound source is called a direct sound, and a sound that is output from the single sound source and then reflected off or diffracted by a non-sound-producing object before reaching the listener's head is called an indirect sound (reflected sound or diffracted sound).

[0501] In this embodiment, the analysis unit 2301 generates an audio signal indicating a reflected sound (indirect sound) and an audio signal indicating a direct sound related to the reflected sound (indirect sound).

[0502] Furthermore, a direct sound related to an indirect sound refers to a direct sound originating from the same sound source as the indirect sound. An indirect sound related to a direct sound refers to an indirect sound originating from the same sound source as the direct sound. Furthermore, a reflected sound is a direct sound related to the reflected sound that is reflected by a reflecting object.

[0503] The audio signal indicating the reflected sound (indirect sound) includes information indicating the audio signal of the direct sound related to the reflected sound (indirect sound).

[0504] In this embodiment, the analysis unit 2301 also generates an audio signal indicating a reflected sound (indirect sound) and an audio signal indicating a direct sound originating from a sound source different from the reflected sound (indirect sound). That is, the analysis unit 2301 generates an audio signal indicating a reflected sound (indirect sound) originating from one sound source, and also generates an audio signal indicating a direct sound originating from another sound source.

[0505] The analysis unit 2301 may store the generated voice signal in a memory included in the analysis unit 2301. The analysis unit 2301 may also generate a plurality of voice signals and store them in the memory.

[0506] Furthermore, the analysis unit 2301 may calculate values ​​relating to the path taken by each of the direct sound and the reflected sound to reach the listening position, the time it takes for the sound to arrive, the volume at the time of arrival, etc., as in embodiment 1. Similarly, the analysis unit 2301 may calculate information indicating the relationship between the direct sound and the reflected sound, such as a value relating to the time difference between the arrival of the direct sound and the reflected sound (the time difference between the direct sound and the reflected sound).

[0507] Examples of the volume upon arrival are the volume upon arrival of reflected sound (lr) and the volume upon arrival of direct sound (ld). The volume upon arrival of direct sound (ld) refers to the volume of direct sound when it arrives at the listening position where the listener is located in the virtual space, or in other words, the volume of direct sound at the listening position. The volume upon arrival of reflected sound (lr) refers to the volume of reflected sound, which is an example of indirect sound, when it arrives at the listening position, or in other words, the volume of indirect sound (reflected sound volume) at the listening position.

[0508] In this embodiment, the audio signal representing the reflected sound and the audio signal representing the direct sound may each include information indicating the volume of the sound represented by the audio signal at the listening position. That is, in this embodiment, the audio signal representing the reflected sound includes information indicating the volume (lr) at the time of arrival of the reflected sound as the indirect sound volume (reflected sound volume). Similarly, the audio signal representing the direct sound includes information indicating the volume (ld) at the time of arrival of the direct sound as the direct sound volume.

[0509] More specifically, the first attribute information includes first volume information indicating the volume (lr) of the reflected sound when it arrives as the volume of the first sound, and the second attribute information includes second volume information indicating the volume (ld) of the direct sound when it arrives as the volume of the second sound.

[0510] The audio signal representing the reflected sound may include information indicating the indirect sound volume (reflected sound volume) and the direct sound volume of the direct sound related to the reflected sound. Similarly, the audio signal representing the direct sound may include information indicating the direct sound volume and the indirect sound volume (reflected sound volume) of the indirect sound (reflected sound) related to the direct sound.

[0511] In addition, the analysis unit 2301 creates first position information and second position information based on the position information of the sound source object contained in the spatial information, the position information of the non-sound-producing object (obstacle object), the position information and physical information of the listener, and the sound data.

[0512] The first position information indicates the position of the first sound, i.e., information for localizing the sound image of the first sound. The second position information indicates the position of the second sound, i.e., information for localizing the sound image of the second sound.

[0513] In this embodiment, the first attribute information includes first location information, and the second attribute information includes second location information.

[0514] The selection unit 2302 may be capable of performing all or part of the processing performed by the selection unit 1302 according to embodiment 1. The selection unit 2302 also determines whether an output signal based on the audio signal created by the analysis unit 2301 is to be output (played) by the playback unit 2303. That is, the selection unit 2302 first determines whether to merge multiple audio signals (e.g., a first audio signal and a second audio signal) created by the analysis unit 2301, and generates a merged merged audio signal if it is determined that the multiple audio signals should be merged. The generated merged audio signal is output by the playback unit 2303.

[0515] The selection unit 2302 includes an acquisition unit 2302a, a determination unit 2302b, and a merging unit 2302c.

[0516] The acquisition unit 2302a acquires an audio signal including attribute information created by the analysis unit 2301 and stored in the memory of the analysis unit 2301. The acquisition unit 2302a acquires, for example, a first audio signal and a second audio signal. The acquisition unit 2302a also acquires a value related to the time difference between a direct sound and a reflected sound calculated by the analysis unit 2301.

[0517] The determination unit 2302b calculates a first arrival direction in which a first sound (reflected sound) arrives at a listening position where a listener is located. The determination unit 2302b calculates a second arrival direction in which a second sound (direct sound) arrives at the listening position. The determination unit 2302b calculates the reflected sound arrival direction as the first arrival direction and the direct sound arrival direction as the second arrival direction using the method described in step S231 of FIG. 23 in the first embodiment.

[0518] Then, the decision unit 2302b decides whether or not to merge the acquired first audio signal and the acquired second audio signal based on an index corresponding to the calculated first arrival direction and second arrival direction.

[0519] When the determination unit 2302b determines to merge the first audio signal and the second audio signal, the merging unit 2302c generates a merged audio signal by merging the first audio signal and the second audio signal. In this case, the merging unit 2302c outputs the generated merged audio signal to the playback unit 2303.

[0520] Furthermore, if it is determined not to merge the first and second audio signals, the merger 2302c does not generate a merged audio signal, and outputs the first and second audio signals to the playback unit 2303.

[0521] The merge sound signal is a signal indicating a merge sound obtained by combining the first sound and the second sound. The volume of the merge sound is the sum of the volumes of the first sound and the second sound. If the first sound is a reflected sound of a dog barking and the second sound is a direct sound of a vehicle running, the merge sound is a combination of the sound of a dog barking and the sound of a vehicle running.

[0522] The playback unit 2303 may be able to perform all or some of the processing performed by the playback unit 1303 according to Embodiment 1. The playback unit 2303 also acquires the merged audio signal, the first audio signal, and the second audio signal output from the selection unit 2302 (more specifically, the merging unit 2302c), and outputs output signals based on the acquired merged audio signal, the first audio signal, and the second audio signal, respectively.

[0523] The playback unit 2303 performs processing such as binaural filtering on each of the acquired merge audio signal, first audio signal, and second audio signal to generate and output an output signal based on each of the acquired merge audio signal, first audio signal, and second audio signal. The binaural filtering is realized, for example, by performing processing (e.g., convolution processing) in which a head-related transfer function is applied to each of the acquired merge audio signal, first audio signal, and second audio signal. In this embodiment, for example, the output signal based on the merge audio signal is generated by applying a head-related transfer function to the merge audio signal.

[0524] Furthermore, the reproduction unit 2303 may perform both binaural filtering and diffusion filtering on each of the merge audio signal, the first audio signal, and the second audio signal output from the selection unit 2302, thereby generating and outputting an output signal based on each. The diffusion filtering is, for example, processing that improves the realism of the merged audio by diffusing the merged audio signal represented by the acquired merged audio signal. The diffusion filtering is processing that uses a filter that realistically simulates the auditory strength of the diffusion of the merged audio represented by the acquired merged audio signal (i.e., simulates the auditory strength of the diffusion of the sound perceived by the listener). The diffusion filtering uses a finite impulse filter and / or an infinite impulse filter.

[0525] An example of the operation of the audio signal processing method performed by the audio signal processing device (more specifically, the rendering unit 2300) according to this embodiment will be described below.

[0526] [Operation Example 1 of Rendering Unit] Fig. 34 is a flowchart showing operation example 1 of the audio signal processing device according to this embodiment. Fig. 34 mainly shows processing executed by the rendering unit 2300 included in the audio signal processing device according to this embodiment. Note that here, explanation of common points with Fig. 8 according to embodiment 1 will be omitted or simplified.

[0527] In this operation example, the first sound is an indirect sound (more specifically, a reflected sound), and the second sound is a direct sound.

[0528] First, the analysis unit 2301 performs an analysis process to analyze an input signal (S101a). For example, the analysis unit 2301 generates a plurality of first audio signals and a plurality of second audio signals, and stores the generated plurality of first audio signals and a plurality of second audio signals in the memory of the analysis unit 2301.

[0529] The selection unit 2302 determines whether or not to merge the first audio signal and the second audio signal, and if it is determined that they should be merged, generates and outputs a merged audio signal (S102a).

[0530] The reproduction unit 2303 acquires the merge audio signal output from the selection unit 2302, and outputs an output signal based on the acquired merge audio signal (S103a).

[0531] Then, the processing performed by the selection unit 2302 and the playback unit 2303 will be described in more detail with reference to FIG.

[0532] 35 is a flowchart showing an example of the operation of the processes performed by the selection unit 2302 and the playback unit 2303 in operation example 1 according to the present embodiment. Note that, here, explanation of commonalities with FIG. 14 according to embodiment 1 will be omitted or simplified.

[0533] First, the selection unit 2302 specifies the reflected sound and the direct sound detected by the analysis unit 2301 (S501). That is, the acquisition unit 2302a of the selection unit 2302 specifies one first sound signal from among the multiple first sound signals created by the analysis unit 2301 and stored in memory, and acquires the specified first sound signal. The acquisition unit 2302a also specifies one second sound signal from among the multiple second sound signals created by the analysis unit 2301 and stored in memory, and acquires the specified second sound signal.

[0534] Then, the determination unit 2302b of the selection unit 2302 calculates a first arrival direction (a direction from which the reflected sound arrives) and calculates a second arrival direction (a direction from which the direct sound arrives). More specifically, the determination unit 2302b calculates the direction from which the reflected sound arrives for the acquired first audio signal and calculates the direction from which the direct sound arrives for the acquired second audio signal.

[0535] The decision unit 2302b decides whether to merge the acquired first audio signal and the acquired second audio signal based on the index corresponding to the calculated first arrival direction and second arrival direction. As an example, the decision unit 2302b performs the following process.

[0536] The determination unit 2302b calculates the arrival angle δ between the reflected sound and the direct sound (S502). That is, the determination unit 2302b calculates the arrival angle δ, which is the angle between the first arrival direction and the second arrival direction. This arrival angle δ is an example of the index.

[0537] Here, the index (arrival angle δ) will be explained.

[0538] As described in the first embodiment, the direction information of the sound source object and the direction information of the listener may be expressed by azimuth (yaw) and elevation (pitch), respectively. Therefore, the arrival angle δ, which is an example of an index, may also be expressed by azimuth (yaw) and elevation (pitch). The index according to this embodiment is an index consisting of an azimuth axis (yaw) and an elevation axis (pitch), i.e., two orthogonal axes.

[0539] The determination unit 2302b compares the calculated arrival angle δ with the discrimination limit T of the arrival angle δ. The discrimination limit T of the arrival angle δ means the minimum angle at which a listener can perceive the movement of a sound image when the sound image moves. A known value can be used for the discrimination limit T (for example, Figure 2.26 of Non-Patent Document 2).

[0540] The discrimination limit T may be set to a predetermined value before this operation example is performed. The discrimination limit T (predetermined value) may be stored in the memory of the analysis unit 2301, for example, and the acquisition unit 2302a acquires the discrimination limit T stored in the memory.

[0541] Then, the decision unit 2302b decides whether to merge the first audio signal and the acquired second audio signal based on the index corresponding to the calculated first and second arrival directions. That is, the decision unit 2302b determines whether the arrival angle δ based on the first and second arrival directions is smaller than the discrimination limit T (S503). In this way, the decision unit 2302b decides whether to merge the first audio signal and the second audio signal.

[0542] If the arrival angle δ is smaller than the discrimination limit T (Yes in S503), the decision unit 2302b decides to merge the first audio signal and the second audio signal. Then, the merging unit 2302c synthesizes (merges) the reflected sound (first sound) and the direct sound (second sound) (S504). That is, the merging unit 2302c generates a merged audio signal by merging the first audio signal and the second audio signal. The merging unit 2302c outputs the generated merged audio signal to the playback unit 2303. In this case, it is preferable that the merging unit 2302c performs processing to disable the first audio signal and the second audio signal before they are merged. The disabled first audio signal and second audio signal are not output by the playback unit 2303.

[0543] Then, the playback unit 2303 plays the synthesized sound (merged sound) (S505). That is, the playback unit 2303 acquires the merged audio signal output from the selection unit 2302, and outputs an output signal based on the acquired merged audio signal. More specifically, the playback unit 2303 outputs an output signal generated by applying a head-related transfer function to the merged audio signal.

[0544] If the arrival angle δ is equal to or greater than the discrimination limit T (No in S503), the decision unit 2302b decides not to merge the first and second audio signals. The merging unit 2302c outputs the first and second audio signals to the playback unit 2303.

[0545] The reproduction unit 2303 reproduces each of the reflected sound and the direct sound (S506). That is, the reproduction unit 2303 acquires the first audio signal and the second audio signal output from the selection unit 2302, and outputs an output signal based on the acquired first audio signal and an output signal based on the acquired second audio signal. More specifically, the reproduction unit 2303 outputs an output signal generated by applying a head-related transfer function to the first audio signal, and outputs an output signal generated by applying a head-related transfer function to the second audio signal.

[0546] Here, a set of a first sound (reflected sound) represented by one first sound signal and a second sound (direct sound) represented by one second sound signal among the plurality of first sound signals and the plurality of second sound signals created by the analysis unit 2301 and stored in memory is defined as one first combination. The selection unit 2302 determines whether the processing of steps S501 to S506 has been performed for all first combinations (S507). That is, the selection unit 2302 determines whether the processing of steps S501 to S506 has been performed for the first sound signals and second sound signals representing the first sound and second sound that are each set of all first combinations.

[0547] If the answer is No in step S507, the process of step S501 is performed again, and if the answer is Yes in step S507, the operation ends.

[0548] [Operation Example 2 of Rendering Unit] In Operation Example 2, the same processing as in Operation Example 1 is performed, except that the processing shown in FIG. 36 is performed instead of the processing shown in FIG.

[0549] 36 is a flowchart showing an example of the operation of the processes performed by the selection unit 2302 and the playback unit 2303 in Operation Example 2 according to the present embodiment. Note that, here, explanation of commonalities with Operation Example 1 according to Embodiment 2 will be omitted or simplified.

[0550] In the second operational example, step S101a shown in FIG. 34 is performed in the same manner as in the first operational example.

[0551] 36 , the selection unit 2302 specifies the direct sound detected by the analysis unit 2301 (S501a). That is, the acquisition unit 2302a of the selection unit 2302 specifies one second audio signal from among the multiple second audio signals created by the analysis unit 2301 and stored in memory, and acquires the specified second audio signal.

[0552] The determination unit 2302b of the selection unit 2302 calculates the second arrival direction (direct sound arrival direction) (S508).

[0553] The selection unit 2302 specifies the reflected sound detected by the analysis unit 2301 (S501aa). That is, the acquisition unit 2302a of the selection unit 2302 specifies one first audio signal from among the multiple first audio signals created by the analysis unit 2301 and stored in memory, and acquires the specified first audio signal.

[0554] The determination unit 2302b of the selection unit 2302 calculates the first arrival direction (the direction from which the reflected sound comes) (S509).

[0555] The determining unit 2302b calculates the arrival angle δ between the reflected sound and the direct sound (S502). That is, the determining unit 2302b calculates the arrival angle δ, which is the angle between the first arrival direction and the second arrival direction.

[0556] The determination unit 2302b determines the discrimination limit T for the arrival angle δ with respect to the arrival direction of the direct sound (S509a). Here, the determination unit 2302b determines the discrimination limit T corresponding to the arrival angle δ with respect to the arrival direction of the direct sound (second arrival direction).

[0557] Here, cases are considered in which a direct sound arrives from above, behind, below, right, left, or the like of the listener. In this case, the discrimination limit T for the arrival angle δ may be determined according to the direction. Here, a database of discrimination limits T for the arrival angle δ predetermined for each direction from which the direct sound arrives is stored in the memory of the analysis unit 2301, and the determination unit 2302b specifies the discrimination limit T corresponding to the calculated second arrival direction.

[0558] In the database, a one-to-one correspondence is defined (determined) between one arrival direction of a direct sound and one value of the discrimination limit T. For example, the determination unit 2302b refers to the database, selects the value of the discrimination limit T that corresponds one-to-one to the arrival direction of the direct sound (second arrival direction), extracts the selected value, and specifies the extracted value as the value of the discrimination limit T.

[0559] A database of discrimination limits T for predetermined arrival angles δ for each direction from which a direct sound comes may be created based on actual measurements in subject experiments.

[0560] On the other hand, when a device or service incorporating the present technology includes a database of head-related transfer functions (so-called SOFA (Spatially Oriented Format for Acoustics) of HRTF (Head Related Impulse Response)), the discrimination limit T of the arrival angle δ for each arrival direction may be determined by calculation processing in accordance with the SOFA of the HRTF. The method for doing so will be described below.

[0561] Fig. 37 is a diagram showing the SOFA of an HRTF according to this embodiment. Fig. 38 is a diagram showing a cone for explaining the relationship between the position where the HRTF according to this embodiment is defined and the listening position.

[0562] In Fig. 37, the listener is positioned at the center of a celestial sphere, and multiple points are arranged on the surface of the celestial sphere. One HRTF is assigned (placed) at each of the multiple points.

[0563] FIG. 38 shows a cone with the position of the listener (listening position) as the vertex, the direction of the incoming direct sound as the perpendicular line, and an angle γ between the perpendicular line and the generatrix.

[0564] When such a cone is applied to the SOFA of the HRTF, the largest γ such that no more than K HRTFs are defined within the cone is set as the discrimination limit T of the arrival angle δ in the direction from which the direct sound arrives. Considering the meaning of the discrimination limit T, it is ideal to set K = 1. However, in reality, by setting N to a relatively large value (e.g., K = 10) so that the discrimination limit T is expressed as a large value, it is possible to increase the opportunities for synthesis of direct sound and reflected sound, thereby reducing the amount of calculation. Such a database can be used in step S509a.

[0565] This will be explained again with reference to Fig. 36. The decision unit 2302b compares the calculated arrival angle δ with the discrimination limit T of the identified arrival angle δ.

[0566] The decision unit 2302b determines whether the arrival angle δ based on the first arrival direction and the second arrival direction is smaller than the discrimination limit T (S503). As a result, the decision unit 2302b decides whether to merge the first audio signal and the second audio signal.

[0567] If the arrival angle δ is smaller than the discrimination limit T (Yes in S503), the decision unit 2302b decides to merge the first sound signal and the second sound signal. The merging unit 2302c combines (merges) the reflected sound (first sound) and the direct sound (second sound) (S504).

[0568] The playback unit 2303 plays back the synthesized sound (merged sound) (S505).

[0569] If the arrival angle δ is equal to or greater than the discrimination limit T (No in S503), the decision unit 2302b decides not to merge the first and second audio signals. The merging unit 2302c outputs the first and second audio signals to the playback unit 2303.

[0570] The reproduction unit 2303 reproduces each of the reflected sound and the direct sound (S506).

[0571] The selection unit 2302 determines whether the processes of steps S501aa to S506 have been performed on all of the multiple first audio signals created by the analysis unit 2301 and stored in memory. That is, the selection unit 2302 determines whether the processes of steps S501aa to S506 have been performed on all reflected sounds represented by the multiple first audio signals created by the analysis unit 2301 and stored in memory (S507a).

[0572] If the answer to step S507a is No, the process of step S501aa is performed again. If the answer to step S507a is Yes, the following process is further performed.

[0573] The selection unit 2302 determines whether or not the processes of steps S501a to S507a have been performed on all of the second audio signals created by the analysis unit 2301 and stored in memory. That is, the selection unit 2302 determines whether or not the processes of steps S501a to S507a have been performed on all of the direct sounds represented by the second audio signals created by the analysis unit 2301 and stored in memory (S507aa).

[0574] If the answer is No in step S507aa, the process of step S501a is performed again. If the answer is Yes in step S507aa, the operation ends.

[0575] [Operation Example 3 of Rendering Unit] In Operation Example 3, the same processing as in Operation Example 1 is performed, except that the processing shown in FIG. 39 is performed instead of the processing shown in FIG.

[0576] 39 is a flowchart showing an example of operation of the processes performed by the selection unit 2302 and the playback unit 2303 in Operation Example 3 according to the present embodiment. Note that, here, explanation of commonalities with Operation Example 1 according to Embodiment 2 will be omitted or simplified.

[0577] In Operation Example 1, the first sound is an indirect sound (more specifically, a reflected sound) and the second sound is a direct sound, but this is not limited to this in Operation Example 3. That is, in Operation Example 2, the first sound and the second sound may both be direct sounds, or both the first sound and the second sound may be indirect sounds, or the first sound may be a direct sound and the second sound may be an indirect sound, or the first sound may be an indirect sound and the second sound may be a direct sound.

[0578] In the third operation example, step S101a shown in FIG. 34 is performed in the same manner as in the first operation example.

[0579] 39, the selection unit 2302 specifies the two sounds detected by the analysis unit 2301 (S501b). That is, the acquisition unit 2302a of the selection unit 2302 specifies two audio signals from among the plurality of first audio signals and the plurality of second audio signals created by the analysis unit 2301 and stored in memory, and acquires the specified two audio signals.

[0580] The two specified sounds may be two first sounds, two second sounds, or one first sound and one second sound. That is, the two specified audio signals may be two first audio signals, two second audio signals, or one first audio signal and one second audio signal.

[0581] The decision unit 2302b of the selection unit 2302 calculates the direction of arrival of each of the two sounds.

[0582] The determination unit 2302b calculates the arrival angle δ of the two sounds (S502b). That is, the determination unit 2302b calculates the arrival angle δ, which is the angle between the arrival direction of one of the two sounds and the arrival direction of the other sound.

[0583] For example, if the two specified sounds are two first sounds, the determination unit 2302b calculates the arrival angle δ, which is the angle between the first arrival direction corresponding to one first sound and the first arrival direction corresponding to the other first sound. Alternatively, if the two specified sounds are two second sounds, the determination unit 2302b calculates the arrival angle δ, which is the angle between the second arrival direction corresponding to one second sound and the second arrival direction corresponding to the other second sound. Alternatively, if the two specified sounds are one first sound and one second sound, the determination unit 2302b calculates the arrival angle δ, which is the angle between the first arrival direction corresponding to one first sound and the second arrival direction corresponding to one second sound.

[0584] The determination unit 2302b compares the calculated arrival angle δ with the discrimination limit T of the arrival angle δ. In this operation example, the discrimination limit T determined as follows is used.

[0585] First, before this operation example is performed, a database indicating the value of the discrimination limit T corresponding to each sound arrival direction is stored in the memory of the analysis unit 2301. The determination unit 2302b selects one of the two specified sounds. The determination unit 2302b references the database to extract a value corresponding to the arrival direction of the selected sound (S510), and determines the extracted value as the value of the discrimination limit T.

[0586] In the database, a one-to-one correspondence is defined (determined) between one sound arrival direction and one value of the discrimination limit T. For example, the determination unit 2302b refers to the database, selects the value of the discrimination limit T that corresponds one-to-one to the first sound arrival direction, extracts the selected value, and determines the extracted value as the value of the discrimination limit T.

[0587] However, without being limited to this, in this operation example as well, the discrimination limit T may be set to a predetermined value in advance before this operation example is performed.

[0588] The decision unit 2302b decides whether to merge the two audio signals based on the calculated index corresponding to the respective arrival directions of the two sounds. That is, the decision unit 2302b determines whether the arrival angle δ based on the two arrival directions is smaller than the discrimination limit T (S503b). Based on this, the decision unit 2302b decides whether to merge the two audio signals.

[0589] If the angle of arrival δ is smaller than the discrimination limit T (Yes in S503b), the decision unit 2302b decides to merge the two audio signals. Then, the merging unit 2302c merges the two sounds (S504b). That is, the merging unit 2302c generates a merged audio signal by merging the two audio signals. The merging unit 2302c outputs the generated merged audio signal to the playback unit 2303. In this case, it is preferable that the merging unit 2302c performs processing to invalidate the two audio signals before they are merged. The invalidated two audio signals are not output by the playback unit 2303.

[0590] If the arrival angle δ is equal to or greater than the discrimination limit T (No in S503b), the decision unit 2302b decides not to merge the two audio signals. The merging unit 2302c outputs the two audio signals to the playback unit 2303. The two sounds represented by the two audio signals correspond to the sounds that were not merged.

[0591] Here, a set of two sounds represented by two audio signals among the plurality of first audio signals and the plurality of second audio signals created by the analysis unit 2301 and stored in memory is defined as one second combination. The selection unit 2302 determines whether the processes of steps S501b to S504b have been performed for all second combinations (S507b). That is, the selection unit 2302 determines whether the processes of steps S501b to S504b have been performed for the two audio signals representing the two sounds that are each a set of all second combinations.

[0592] If the answer is No in step S507b, the process of step S501b is performed again. If the answer is Yes in step S507b, the following process is performed.

[0593] The reproduction unit 2303 applies a head-related transfer function based on the arrival direction to the non-merged sound and the merged sound (merged sound) (S505b). More specifically, the reproduction unit 2303 outputs an output signal generated by applying, to one of the two audio signals determined by the determination unit 2302b not to be merged, a head-related transfer function based on the arrival direction of the sound indicated by the one audio signal. Furthermore, the reproduction unit 2303 outputs an output signal generated by applying, to the other of the two audio signals determined by the determination unit 2302b not to be merged, a head-related transfer function based on the arrival direction of the sound indicated by the other audio signal.

[0594] That is, for example, when the two audio signals determined not to be merged are one first audio signal and one second audio signal, the reproduction unit 2303 performs the following process: The reproduction unit 2303 outputs an output signal generated by applying a head-related transfer function based on the first direction of arrival to the first audio signal, and an output signal generated by applying a head-related transfer function based on the second direction of arrival to the second audio signal.

[0595] Then, the playback unit 2303 outputs an output signal generated by applying a head-related transfer function based on the merge sound arrival direction from which the merge sound indicated by the merge audio signal arrives at the listening position to the merge audio signal generated by the merge unit 2302c.

[0596] The reproduction unit 2303 may select a head-related transfer function to be applied by using a database (e.g., SOFA of HRIR) that indicates head-related transfer functions defined for each sound arrival direction. In this database, one sound arrival direction and one head-related transfer function are defined (determined) to correspond one-to-one.

[0597] For example, the reproduction unit 2303 may refer to the database, select a head-related transfer function that corresponds one-to-one with the first arrival direction, and apply the selected head-related transfer function to the first audio signal to generate and output the output signal. Note that similar processing is performed for the second arrival direction and the merge sound arrival direction.

[0598] Here, the merging sound arrival direction used in step S505b will be described. In the description of the merging sound arrival direction, the two sounds specified in step S501b will be described as one first sound and one second sound.

[0599] Since one first sound and one second sound are specified in step S501b, the acquiring unit 2302a acquires the first audio signal and the second audio signal.

[0600] As described above, the first attribute information included in the acquired first audio signal includes first position information indicating the position of the first sound and first volume information indicating the volume of the first sound, and the second attribute information included in the acquired second audio signal includes second position information indicating the position of the second sound and second volume information indicating the volume of the second sound.

[0601] The merging unit 2302c determines the direction from which the merge sound indicated by the merge audio signal arrives at the listening position, based on the first position information, the first volume information, and the second position information and the second volume information. A specific example will be described with reference to FIG. 40 .

[0602] FIG. 40 is a diagram for explaining the direction from which a merging sound comes, which is used in the operation example 3 according to the present embodiment.

[0603] The first position information and the second position information are information that indicates the positions of the first sound and the second sound in polar coordinates. For example, as shown in Fig. 40, the first position information indicates that the polar coordinate argument for the position of the first sound is 135°, and the second position information indicates that the polar coordinate argument for the position of the second sound is 45°. Note that when the first position information and the second position information are indicated in polar coordinates, the argument for the sound corresponds to the direction from which the sound arrives to the listener.

[0604] Also, as shown in Figure 40, for example, the first volume information indicates that the volume of the first sound (volume at the listening position) is 1.0, and the second volume information indicates that the volume of the second sound (volume at the listening position) is 3.0.

[0605] The merging unit 2302c first determines the position of the merging sound as follows.

[0606] The merging unit 2302c regards the volume of the first sound indicated by the first volume information as a weight corresponding to the position of the first sound indicated by the first position information. The merging unit 2302c regards the volume of the second sound indicated by the second volume information as a weight corresponding to the position of the second sound indicated by the second position information. In this case, the merging unit 2302c determines that the position of the merged sound is the position of the center of gravity of the first sound and the second sound.

[0607] That is, the merging unit 2302c determines the position of the center of gravity of the first sound and the second sound as the position of the merging sound (more specifically, the position where the merging sound is generated). In FIG. 40, the position of the center of gravity is calculated to have an angle of deviation (direction of arrival) of 67.5°. Furthermore, the volume of the merging sound is calculated to be 4.0, since it is the sum of the volume of the first sound and the volume of the second sound. In this way, the merging unit 2302c determines the position of the merging sound.

[0608] The merging unit 2302c then determines the direction from which the merge sound comes, using the determined position of the merge sound and the position information of the avatar (listener) and the orientation information D1 of the avatar (listener) included in the input signal. In other words, the merging unit 2302c determines the direction from which the merge sound comes, which is the direction from the determined position of the merge sound (the position of the center of gravity of the first sound and the second sound) toward the listening position.

[0609] As described above, in this embodiment, the merging unit 2302c determines the direction from which the merging sound comes based on the first position information and the first volume information, and the second position information and the second volume information.

[0610] Furthermore, in the above description, the volume is used to determine the position of the center of gravity, but this is not limiting, and loudness may be used instead of the volume.

[0611] Then, in step S505b, the playback unit 2303 outputs an output signal generated by applying a head-related transfer function based on the merge sound arrival direction to the merge audio signal.

[0612] In step S510, the determination unit 2302b may select one of the two specified sounds in accordance with the following conditions: For example, if one of the two specified sounds is a direct sound, the determination unit 2302b selects the direct sound. For example, if both of the two specified sounds are direct sounds or indirect sounds (reflected sounds), the determination unit 2302b selects a sound with a high volume, a high loudness, a sound coming from a stationary direction (i.e., a sound whose sound source is not moving), or a sound coming from a direction close to the front of the listener. The determination unit 2302b may determine a value according to the direction of arrival of a merged sound obtained by merging two sounds as the value of the discrimination limit T.

[0613] [Operation Example 4 of the Rendering Unit] In Operation Example 4, the same processing as in Operation Example 1 is performed, except that the processing shown in FIG. 41 is performed instead of the processing shown in FIG.

[0614] 41 is a flowchart showing an example of the operation of the processes performed by the selection unit 2302 and the playback unit 2303 in Operation Example 4 according to the present embodiment. Note that, here, explanation of commonalities with Operation Example 3 according to Embodiment 2 will be omitted or simplified.

[0615] In the fourth operational example, step S101a shown in FIG. 34 is performed in the same manner as in the first operational example.

[0616] Then, as shown in FIG. 41, the selection unit 2302 designates the two sounds detected by the analysis unit 2301 (S501b).

[0617] The determination unit 2302b of the selection unit 2302 calculates the position where the merge sound will be generated when two specified sounds are merged, and the direction from which the merge sound will be generated to the listening position (merging sound arrival direction).

[0618] The method for calculating the position where the merging sound is generated and the direction from which the merging sound comes is as described in the third operation example.

[0619] Furthermore, the determining unit 2302b calculates the direction of arrival of each of the two designated sounds.

[0620] The determination unit 2302b calculates the arrival angles (arrival angles δ1 and δ2) of the merged sound when the two sounds are merged and each of the two specified sounds (S502c). That is, the determination unit 2302b calculates the arrival angle δ1, which is the angle between the arrival direction of the merged sound and the arrival direction of one of the two specified sounds, and calculates the arrival angle δ2, which is the angle between the arrival direction of the merged sound and the arrival direction of the other of the two specified sounds.

[0621] The determination unit 2302b compares the calculated arrival angles δ1 and δ2 with the arrival angle discrimination limit T. In this operation example, the discrimination limit T determined as follows is used.

[0622] A database indicating the value of discrimination limit T corresponding to each sound arrival direction is stored in the memory of the analysis unit 2301. The determination unit 2302b refers to the database to extract a value corresponding to the arrival direction (merging sound arrival direction) from the position where the merged sound is generated to the listening position when two specified sounds are merged (S511), and determines the extracted value as the value of discrimination limit T.

[0623] The decision unit 2302b decides whether to merge the two audio signals based on the index corresponding to the calculated arrival direction of each of the two sounds. That is, the decision unit 2302b determines whether the calculated arrival angle δ1 is smaller than the discrimination limit T and whether the calculated arrival angle δ2 is smaller than the discrimination limit T (S503c). In this way, the decision unit 2302b decides whether to merge the two audio signals.

[0624] If the arrival angle δ1 is smaller than the discrimination limit T and the arrival angle δ2 is smaller than the discrimination limit T (Yes in S503c), the decision unit 2302b decides to merge the two audio signals. Then, the merging unit 2302c merges the two sounds (S504b).

[0625] If the arrival angle δ1 is equal to or greater than the discrimination limit T or if the arrival angle δ2 is equal to or greater than the discrimination limit T (No in S503c), the decision unit 2302b decides not to merge the two audio signals.

[0626] The selection unit 2302 determines whether or not the processes of steps S501b to S504b have been performed for all second combinations (S507b).

[0627] If the answer is No in step S507b, the process of step S501b is performed again. If the answer is Yes in step S507b, the following process is performed.

[0628] The playback unit 2303 applies a head-related transfer function to the non-merged sound and the merged sound (merged sound) according to the direction of arrival (S505b).

[0629] [Operation Example 5 of Rendering Unit] In Operation Example 5, the same processing as in Operation Example 1 is performed, except that the processing shown in FIG. 42 is performed instead of the processing shown in FIG.

[0630] 42 is a flowchart showing an example of operation of the processes performed by the selection unit 2302 and the playback unit 2303 in Operation Example 5 according to the present embodiment. Note that here, explanations of commonalities with Operation Examples 1 and 4 according to Embodiment 2 will be omitted or simplified.

[0631] In the fifth operational example, step S101a shown in FIG. 34 is performed in the same manner as in the first operational example.

[0632] The selection unit 2302 specifies the reflected sound and the direct sound detected by the analysis unit 2301 (S501).

[0633] The determining unit 2302b calculates the arrival angle δ between the reflected sound and the direct sound (S502).

[0634] The determining unit 2302b determines whether the arrival angle δ based on the first arrival direction and the second arrival direction is smaller than the discrimination limit T (S503). The discrimination limit T may be acquired by the acquiring unit 2302a, similar to the discrimination limit T of the arrival angle δ in the first operational example.

[0635] If the arrival angle δ is smaller than the discrimination limit T (Yes in S503), the merging unit 2302c combines (merges) the reflected sound (first sound) and the direct sound (second sound) (S504). In this case, it is preferable that the merging unit 2302c performs processing to invalidate the first audio signal and the second audio signal before they are merged.

[0636] If the arrival angle δ is equal to or greater than the discrimination limit T (No in S503), the decision unit 2302b decides not to merge the first and second audio signals. The merging unit 2302c outputs the first and second audio signals to the playback unit 2303.

[0637] The selection unit 2302 determines whether the processes of steps S501 to S504 have been performed for all first combinations (S507).

[0638] If the answer is No in step S507, the process of step S501 is performed again. If the answer is Yes in step S507, the following process is performed.

[0639] The reproduction unit 2303 applies a head-related transfer function corresponding to the arrival direction of the direct sound and the non-invalidated reflected sound to the direct sound and the non-invalidated reflected sound (S505d). More specifically, the reproduction unit 2303 outputs an output signal generated by applying a head-related transfer function corresponding to the arrival direction of the second sound indicated by the second audio signal, to the second audio signal indicating the direct sound. The reproduction unit 2303 also outputs an output signal generated by applying a head-related transfer function corresponding to the arrival direction of the reflected sound indicated by the first audio signal, to the first audio signal that has not been subjected to the invalidation process by the merging unit 2302c.

[0640] As in the third operation example, the reproduction unit 2303 may use a database that indicates head-related transfer functions corresponding to the respective directions from which sounds arrive.

[0641] In the fifth operational example, a reflected sound is used, but the same processing is performed even if an indirect sound is used instead of the reflected sound.

[0642] [Operation Example 6 of the Rendering Unit] Operation example 6 is the same as operation example 5, except that the discrimination limit T is extracted as follows:

[0643] 43 is a flowchart showing an example of operation of the processes performed by the selection unit 2302 and the playback unit 2303 in Operation Example 6 according to the present embodiment. Note that, here, explanation of commonalities with Operation Example 5 according to Embodiment 2 will be omitted or simplified.

[0644] In the fifth operational example, step S101a shown in FIG. 34 is performed in the same manner as in the first operational example.

[0645] Steps S501 and S502 are carried out.

[0646] In this operation example, as in operation example 3, a database indicating the value of the discrimination limit T corresponding to each direction from which a sound arrives is stored in the memory of the analysis unit 2301. The determination unit 2302b refers to the database to extract a value corresponding to the direction from which the direct sound arrives (S512), and determines the extracted value as the value of the discrimination limit T.

[0647] The discrimination limit T determined as above is used to perform step S503.

[0648] Steps S504, S507 and S505d are carried out.

[0649] Note that a database is used in each of step S512 and step S505d. The threshold value indicated by the database used in step S512 and the head-related transfer function indicated by the database used in step S505d may be stored in a common address area for each sound arrival direction.

[0650] [Operation Example 7 of Rendering Unit] In Operation Example 7, the same processing as in Operation Example 1 is performed, except that the processing shown in FIG. 44 is performed instead of the processing shown in FIG.

[0651] 44 is a flowchart showing an example of operation of the processes performed by the selection unit 2302 and the playback unit 2303 in Operation Example 7 according to the present embodiment. Note that here, explanations of commonalities with Operation Examples 1, 2, and 6 according to Embodiment 2 will be omitted or simplified.

[0652] In the seventh operational example, step S101a shown in FIG. 34 is performed in the same manner as in the first operational example.

[0653] The selection unit 2302 designates the reflected sound detected by the analysis unit 2301 (S501aa). The reflected sound (first audio signal) designated at this time may be referred to as the first reflected sound.

[0654] The selection unit 2302 designates the direct sound detected by the analysis unit 2301 (S501a).

[0655] The determining unit 2302b calculates the arrival angle δ between the reflected sound and the direct sound (S502).

[0656] In this operation example, as in operation example 3, a database indicating the value of the discrimination limit T corresponding to each sound arrival direction is stored in the memory of the analysis unit 2301. The determination unit 2302b refers to the database to extract a value corresponding to the arrival direction of the reflected sound or the arrival direction of the direct sound (S513), and determines the extracted value as the value of the discrimination limit T.

[0657] The decision unit 2302b determines whether the arrival angle δ based on the first arrival direction and the second arrival direction is smaller than the discrimination limit T (S503). In this operation example, for example, a value according to the arrival direction of the reflected sound is used as the discrimination limit T.

[0658] If the arrival angle δ is smaller than the discrimination limit T (Yes in S503), the determining unit 2302b determines the direct sound (second sound) as the sound to be synthesized (merged) (S514).

[0659] If the arrival angle δ is equal to or greater than the discrimination limit T (No in S503), the decision unit 2302b decides that the direct sound (second sound) is not to be synthesized (merged). In other words, the decision unit 2302b decides not to merge the first audio signal and the second audio signal.

[0660] The selection unit 2302 determines whether the processes of steps S501a to S514 have been performed on all of the second audio signals created by the analysis unit 2301 and stored in memory. That is, the selection unit 2302 determines whether the processes of steps S501a to S514 have been performed on all of the direct sounds represented by the second audio signals created by the analysis unit 2301 and stored in memory (S515).

[0661] If the answer is No in step S515, the process of step S501a is performed again. If the answer is Yes in step S515, the following process is further performed.

[0662] The merging unit 2302c combines the direct sound determined to be merged in step S514 with the reflected sound (first reflected sound) specified in step S501aa (S516). That is, the merging unit 2302c generates a merged audio signal by merging the second audio signal indicating the direct sound determined to be merged with the first audio signal indicating the first reflected sound. The merging unit 2302c outputs the generated merged audio signal to the playback unit 2303. In this case, it is preferable that the merging unit 2302c perform processing to disable the first audio signal before being merged.

[0663] Here, when there are multiple direct sounds determined to be merged, the merging unit 2302c may perform the following process, for example.

[0664] The merging unit 2302c may merge each of the multiple direct sounds with the first reflected sound to generate merged audio signals in the same number as the multiple direct sounds. Furthermore, the merging unit 2302c may merge the direct sound that is closest to the first reflected sound among the multiple direct sounds with the first reflected sound. The merging unit 2302c may merge the direct sound with the loudest volume among the multiple direct sounds with the first reflected sound. The merging unit 2302c may merge the first reflected sound with the direct sound whose sound generation position is stationary among the multiple direct sounds. The merging unit 2302c may merge the first reflected sound with the direct sound whose sound generation position is most visible to the listener among the multiple direct sounds.

[0665] The selection unit 2302 determines whether the processes of steps S501aa to S516 have been performed on all of the multiple first audio signals created by the analysis unit 2301 and stored in memory. That is, the selection unit 2302 determines whether the processes of steps S501aa to S516 have been performed on all reflected sounds represented by the multiple first audio signals created by the analysis unit 2301 and stored in memory (S517).

[0666] If the result of step S517 is No, the process of step S501aa is performed again. When the process of step S501aa is performed again, a reflected sound is specified again, and the reflected sound (first audio signal) specified at this time becomes the second reflected sound.

[0667] After that, when step S516 is performed again, the merging unit 2302c merges the direct sound determined as the merge target in step S514 with the reflected sound (second reflected sound) specified again in step S501aa. The same process is performed for the third and subsequent times.

[0668] If the answer is Yes in step S517, step S505d is performed.

[0669] In the seventh operational example, a reflected sound is used, but the same processing is performed even if an indirect sound is used instead of a reflected sound.

[0670] In the operational examples 3, 4, 6, and 7, the discrimination limit T may be changed according to the following.

[0671] For example, the discrimination limit T may be changed according to the amount of movement of the listener, ie, the amount of rotation of the listener's head.

[0672] When the head is rotating at high speed, the merging of the first sound and the second sound, for example, does not have much effect on the listener's perception compared to when the listener is stationary. Therefore, when the listener moves a lot, the discrimination limit T is set (determined) to be larger compared to when the listener is stationary, thereby reducing the amount of processing without impairing the listener's listening experience. The amount of movement of the listener may be acquired by a sensor included in the audio signal processing device, or may be acquired by the audio signal processing device from an external sensor.

[0673] Furthermore, for example, the discrimination limit T may be changed depending on whether or not a video is displayed.

[0674] When a video is presented to the listener simultaneously with sound output, even if the acoustic effect for perceiving the direction of arrival is not precise, the visual perception of the direction of arrival can be followed. Therefore, when a video is presented, the discrimination limit T is set (determined) to be larger than when no video is presented, thereby reducing the amount of processing without impairing the listener's listening experience.

[0675] The present embodiment can be summarized as follows.

[0676] The audio signal processing method according to this embodiment is an audio signal processing method executed by an audio signal processing device, and includes an acquisition step, a determination step, a merging step, and a reproduction step.

[0677] The acquiring step acquires a first audio signal indicating a first sound, the first audio signal including first attribute information specifying an attribute of the first audio signal, and a second audio signal indicating a second sound, the second audio signal including second attribute information specifying an attribute of the second audio signal. The determining step determines whether to merge the acquired first audio signal and the acquired second audio signal based on an index corresponding to a first arrival direction from which the first sound arrives at a listening position where a listener is located and a second arrival direction from which the second sound arrives at the listening position.

[0678] The merging step generates a merged audio signal by merging the first audio signal and the second audio signal when it is determined to merge the first audio signal and the second audio signal, and the reproduction step outputs an output signal based on the generated merged audio signal.

[0679] As a result, it is determined whether the first audio signal and the second audio signal are merged based on an index corresponding to the first arrival direction of the first sound and the second arrival direction of the second sound. If it is determined that the first audio signal and the second audio signal are merged, the reproducing step outputs an output signal based on the merged audio signal. If it is determined that the first audio signal and the second audio signal are not merged, the reproducing step outputs an output signal based on the first audio signal and an output signal based on the second audio signal. Compared to when both the output signal based on the first audio signal and the output signal based on the second audio signal are output, when the output signal based on the merged audio signal is output, the number of output signals to be output is reduced, thereby reducing the amount of calculation and the calculation load. In other words, it is possible to realize an audio signal processing method that can reduce the amount of calculation and the calculation load.

[0680] Furthermore, the index is, for example, the arrival angle δ. When the arrival angle δ is smaller than the discrimination limit T, the determining step determines to merge the first audio signal and the second audio signal. A case where the arrival angle δ is smaller than the discrimination limit T corresponds to a case where the first sound and the second sound arrive at the listener from the same direction or approximately the same direction. In this case, the first sound (reflected sound) and the second sound (direct sound) are combined, that is, the first audio signal and the second audio signal are merged to generate a single merged audio signal. Even when this processing is performed, the listener is unlikely to feel uncomfortable. Furthermore, as described above, since the number of output signals is reduced, it is possible to realize an audio signal processing method that can more appropriately reduce the amount of calculation and the calculation load.

[0681] In the audio signal processing method according to this embodiment, the index is an index consisting of two orthogonal axes.

[0682] This allows the use of simple indices, i.e., does not require a large amount of calculation and a large calculation load compared to when complex indices are used, thereby realizing an audio signal processing method that can reduce the amount of calculation and the calculation load.

[0683] In the audio signal processing method according to the present embodiment, the first attribute information included in the acquired first audio signal includes first position information indicating the position of the first sound and first volume information indicating the volume of the first sound. The second attribute information included in the acquired second audio signal includes second position information indicating the position of the second sound and second volume information indicating the volume of the second sound. In the merging step, a merge sound arrival direction in which the merge sound indicated by the merge audio signal arrives at the listening position is determined based on the first position information and the first volume information and the second position information and the second volume information.

[0684] As a result, in the merging step, the direction from which the merging sound comes can be determined based on the first position information and the first volume information and the second position information and the second volume information.

[0685] In the audio signal processing method according to the present embodiment, in the merging step, the volume of the first sound indicated by the first volume information is regarded as a weight corresponding to the position of the first sound indicated by the first position information. Furthermore, in the merging step, if the volume of the second sound indicated by the second volume information is regarded as a weight corresponding to the position of the second sound indicated by the second position information, the following processing is performed. In the merging step, the position of the merged sound is determined to be the position of the center of gravity of the first sound and the second sound, and the direction from the determined position of the center of gravity toward the listening position is determined as the arrival direction of the merged sound.

[0686] This allows the merge sound arrival direction to be determined based on the positions of the centers of gravity of the first sound and the second sound, which means that the merge sound arrival direction can be determined easily. In other words, since determining the merge sound arrival direction does not require a large amount of calculation and a large calculation load, it is possible to realize an audio signal processing method that can reduce the amount of calculation and the calculation load.

[0687] The audio signal processing method according to this embodiment is an audio signal processing method executed by an audio signal processing device, and includes an acquisition step, a merging step, and a reproduction step.

[0688] The acquisition step acquires a first audio signal indicating a first sound, the first audio signal including first attribute information that identifies an attribute of the first audio signal, and a second audio signal indicating a second sound, the second audio signal including second attribute information that identifies an attribute of the second audio signal. The merging step generates a merged audio signal by merging the first audio signal and the second audio signal when it is determined to merge the acquired first audio signal and the acquired second audio signal. The reproduction step outputs an output signal based on the generated merged audio signal.

[0689] The first attribute information included in the acquired first audio signal includes first position information indicating the position of the first sound and first volume information indicating the volume of the first sound. The second attribute information included in the acquired second audio signal includes second position information indicating the position of the second sound and second volume information indicating the volume of the second sound. In the merging step, a merge sound arrival direction in which the merge sound indicated by the merge audio signal arrives at a listening position where a listener is located is determined based on the first position information and the first volume information and the second position information and the second volume information.

[0690] As a result, if it is determined that the first audio signal and the second audio signal are merged, the reproducing step outputs an output signal based on the merged audio signal. If it is determined that the first audio signal and the second audio signal are not merged, the reproducing step outputs an output signal based on the first audio signal and an output signal based on the second audio signal. Compared to when both the output signal based on the first audio signal and the output signal based on the second audio signal are output, when the output signal based on the merged audio signal is output, the number of output signals to be output is reduced, thereby reducing the amount of calculation and the calculation load. In other words, an audio signal processing method that can reduce the amount of calculation and the calculation load can be realized.

[0691] In the audio signal processing method of this embodiment, when it is decided to merge the first audio signal and the second audio signal, in the playback step, an output signal generated by applying a head-related transfer function based on the determined direction from which the merge sound arrives to the generated merge audio signal is output.

[0692] This makes it possible to realize an audio signal processing method that allows the listener to hear merged sounds with a more realistic feel.

[0693] In the audio signal processing method according to the present embodiment, when it is determined not to merge the first audio signal and the second audio signal, the reproduction step outputs an output signal generated by applying a head-related transfer function based on the first arrival direction to the first audio signal, and the reproduction step outputs an output signal generated by applying a head-related transfer function based on the second arrival direction to the second audio signal.

[0694] This makes it possible to realize an audio signal processing method that allows the listener to hear the first sound and the second sound with a more realistic feel.

[0695] The computer program according to this embodiment is a computer program for causing a computer to execute the above-described audio signal processing method.

[0696] This allows the computer to execute the above-described audio signal processing method in accordance with the computer program.

[0697] The audio signal processing device according to this embodiment includes an acquisition unit 2302 a , a determination unit 2302 b , a merging unit 2302 c , and a reproduction unit 2303 .

[0698] The acquisition unit 2302a acquires a first audio signal indicating a first sound, the first audio signal including first attribute information that specifies an attribute of the first audio signal, and a second audio signal indicating a second sound, the second audio signal including second attribute information that specifies an attribute of the second audio signal. The decision unit 2302b decides whether to merge the acquired first audio signal and the acquired second audio signal, based on an index corresponding to a first arrival direction from which the first sound arrives at a listening position where a listener is located and a second arrival direction from which the second sound arrives at the listening position.

[0699] When it is determined that the first audio signal and the second audio signal are to be merged, the merger unit 2302c generates a merged audio signal by merging the first audio signal and the second audio signal. The reproduction unit 2303 outputs an output signal based on the generated merged audio signal.

[0700] As a result, it is determined whether the first audio signal and the second audio signal are merged based on an index corresponding to the first arrival direction of the first sound and the second arrival direction of the second sound. If it is determined that the first audio signal and the second audio signal are merged, the reproduction unit 2303 outputs an output signal based on the merged audio signal. If it is determined that the first audio signal and the second audio signal are not merged, the reproduction unit 2303 outputs an output signal based on the first audio signal and an output signal based on the second audio signal. Compared to when both the output signal based on the first audio signal and the output signal based on the second audio signal are output, when the output signal based on the merged audio signal is output, the number of output signals to be output is reduced, thereby reducing the amount of calculation and the calculation load. In other words, it is possible to realize an audio signal processing device that can reduce the amount of calculation and the calculation load.

[0701] Furthermore, the index is, for example, the arrival angle δ. When the arrival angle δ is smaller than the discrimination limit T, the decision unit 2302b decides to merge the first audio signal and the second audio signal. When the arrival angle δ is smaller than the discrimination limit T, this corresponds to a case where the first sound and the second sound arrive at the listener from the same direction or approximately the same direction. In this case, the first sound (reflected sound) and the second sound (direct sound) are combined, that is, the first audio signal and the second audio signal are merged to generate a single merged audio signal. Even when this processing is performed, the listener is unlikely to feel uncomfortable. Furthermore, as described above, since the number of output signals is reduced, it is possible to realize an audio signal processing device that can more appropriately reduce the amount of calculation and the calculation load.

[0702] (Embodiment 3) The following describes embodiment 3. The following description will focus on the differences from embodiments 1 and 2, and the description of commonalities will be omitted or simpli...

Claims

1. A method for processing audio signals performed by an audio signal processing device, Acquisition steps include acquiring a first audio signal that represents a first tone and includes first attribute information that identifies the attributes of the first audio signal, and a second audio signal that represents a second tone and includes second attribute information that identifies the attributes of the second audio signal, A decision step of determining whether or not to merge the acquired first audio signal and the acquired second audio signal based on an index corresponding to a first arrival direction in which the first sound arrives at the listening position where the listener is located and a second arrival direction in which the second sound arrives at the listening position, If it is decided to merge the first audio signal and the second audio signal, a merge step is given to generate a merged audio signal obtained by merging the first audio signal and the second audio signal, The process includes a playback step of outputting an output signal based on the generated merged audio signal, Audio signal processing method.

2. The aforementioned index is an index consisting of two orthogonal axes. The audio signal processing method according to claim 1.

3. The first attribute information included in the acquired first audio signal includes first position information indicating the position of the first sound and first volume information indicating the volume of the first sound. The second attribute information included in the acquired second audio signal includes second position information indicating the position of the second sound and second volume information indicating the volume of the second sound. In the merging step, the direction in which the merged sound indicated by the merged audio signal arrives at the listening position is determined based on the first position information and the first volume information and the second position information and the second volume information. The audio signal processing method according to claim 1.

4. In the aforementioned merge step, When the volume of the first sound indicated by the first volume information is considered to be the weight corresponding to the position of the first sound indicated by the first position information, and the volume of the second sound indicated by the second volume information is considered to be the weight corresponding to the position of the second sound indicated by the second position information, The position of the merged sound is determined to be the position of the center of gravity of the first and second sounds. The direction from the determined position of the center of gravity toward the listening position is determined as the direction of arrival of the merged sound. The audio signal processing method according to claim 3.

5. A method for processing audio signals performed by an audio signal processing device, Acquisition steps include acquiring a first audio signal that represents a first tone and includes first attribute information that identifies the attributes of the first audio signal, and a second audio signal that represents a second tone and includes second attribute information that identifies the attributes of the second audio signal, If it is decided to merge the acquired first audio signal and the acquired second audio signal, a merge step is performed to generate a merged audio signal obtained by merging the first audio signal and the second audio signal, The process includes a playback step of outputting an output signal based on the generated merged audio signal, The first attribute information included in the acquired first audio signal includes first position information indicating the position of the first sound and first volume information indicating the volume of the first sound. The second attribute information included in the acquired second audio signal includes second position information indicating the position of the second sound and second volume information indicating the volume of the second sound. In the merging step, based on the first position information and the first volume information and the second position information and the second volume information, the direction of arrival of the merged sound is determined so that the merged sound indicated by the merged audio signal arrives at the listening position, which is the position of the listener. Audio signal processing method.

6. If it is decided to merge the first audio signal and the second audio signal, In the playback step, the output signal is output by applying a head-related transfer function based on the determined direction of arrival of the merged sound to the generated merged audio signal. The audio signal processing method according to claim 3.

7. If it is decided not to merge the first audio signal and the second audio signal, In the playback step, the output signal generated by applying a head-level transfer function based on a first direction of arrival to the first audio signal, and the output signal generated by applying a head-level transfer function based on a second direction of arrival to the second audio signal are output. The audio signal processing method according to claim 1.

8. A method for processing audio signals performed by an audio signal processing device, An acquisition step of acquiring M audio signals (where M is an integer of 2 or more) that represent a predetermined sound and include attribute information that identifies the attributes of the audio signal, A decision step of determining whether or not to merge N audio signals (where N is an integer between 1 and M) from the acquired M audio signals, based on the direction of arrival of each of the M predetermined sounds arriving at the listening position where the listener is located, When it is decided to merge N audio signals, a merge step is made to generate a merged audio signal obtained by merging the N audio signals, The process includes a playback step of outputting an output signal based on the generated merged audio signal, Audio signal processing method.

9. Each of the M acquired audio signals contains attribute information including predetermined sound position information indicating the position of the predetermined sound and predetermined sound volume information indicating the volume of the predetermined sound. In the merging step, based on the N predetermined sound position information and the N predetermined sound volume information, the direction in which the merged sound indicated by the merged audio signal arrives at the listening position is determined. The audio signal processing method according to claim 8.

10. In the aforementioned merge step, When the volume of the predetermined sound indicated by the predetermined sound volume information is considered to be the weight corresponding to the position of the predetermined sound indicated by the predetermined sound position information for that predetermined sound, The position of the merged sound is determined to be the position of the centroid of the N predetermined sounds. The direction from the determined position of the center of gravity toward the listening position is determined as the direction of arrival of the merged sound. The audio signal processing method according to claim 9.

11. A method for processing audio signals performed by an audio signal processing device, Acquisition steps include acquiring a first audio signal that represents a first tone and includes first attribute information that identifies the attributes of the first audio signal, and a second audio signal that represents a second tone and includes second attribute information that identifies the attributes of the second audio signal, A decision step of determining whether or not to merge the acquired first audio signal and the acquired second audio signal based on the first direction of arrival of the first sound to the listening position where the listener is located and the second direction of arrival of the second sound to the listening position, If it is decided to merge the first audio signal and the second audio signal, a merge step is given to generate a merged audio signal obtained by merging the first audio signal and the second audio signal, The process includes a playback step of outputting an output signal based on the generated merged audio signal, In the determination step, from among a plurality of arrival direction information that each indicates the direction of arrival of the sound when it arrives at the listening position and the head-related transfer function based on the direction of arrival, one arrival direction information corresponding to the first direction of arrival and one arrival direction information corresponding to the second direction of arrival are determined. In the merge step, if one of the direction of arrival information corresponding to the determined first direction of arrival and one of the direction of arrival information corresponding to the determined second direction of arrival are the same, the merged audio signal is generated. In the aforementioned playback step, The generated output signal is output by applying the head-direction transfer function indicated by one of the arrival direction pieces corresponding to the first arrival direction to the generated merged audio signal. When one of the arrival direction information corresponding to the determined first arrival direction and one of the arrival direction information corresponding to the determined second arrival direction are different, the system outputs an output signal generated by applying the head-level transfer function indicated by the one of the arrival direction information corresponding to the first arrival direction to the first audio signal, and an output signal generated by applying the head-level transfer function indicated by the one of the arrival direction information corresponding to the second arrival direction to the second audio signal. Audio signal processing method.

12. A computer program for causing a computer to execute the audio signal processing method described in any one of claims 1 to 11.

13. An acquisition unit that acquires a first audio signal representing a first tone and including first attribute information that identifies the attributes of the first audio signal, and a second audio signal representing a second tone and including second attribute information that identifies the attributes of the second audio signal, A determination unit determines whether or not to merge the acquired first audio signal and the acquired second audio signal based on an index corresponding to a first arrival direction in which the first sound arrives at the listening position where the listener is located, and a second arrival direction in which the second sound arrives at the listening position. When it is decided to merge the first audio signal and the second audio signal, a merging unit generates a merged audio signal obtained by merging the first audio signal and the second audio signal, The system includes a playback unit that outputs an output signal based on the generated merged audio signal, Audio signal processing device.