Acoustic processing device, acoustic processing method, and program

Abstract

An acoustic processing device (information processing device (101)) comprises: an acquisition unit (111) that acquires sound information which includes an acoustic signal and information pertaining to the position of a sound source object in a three-dimensional sound field; a characteristic acquisition unit (115) that acquires information pertaining to listening characteristics of a user (99); and a reduction processing unit (132) that, when generating an output sound signal from the acoustic signal which is included in the acquired sound information, generates an output sound signal which does not include a signal of at least one sound by reducing the signal of the at least one sound on the basis of the acquired information pertaining to the listening characteristics of the user (99).

Description

Sound processing device, sound processing method, and program

[0001] The present disclosure relates to a sound processing device, a sound processing method, and a program.

[0002] Conventionally, there are known techniques for reproducing sound in a virtual three-dimensional space to allow a user to perceive stereoscopic sound (see, for example, Patent Document 1). Furthermore, in order to perceive sound as if it is coming from a sound source object to the user in such a three-dimensional space, processing is required to generate output sound information from original sound information. In particular, reproducing stereoscopic sound in response to the user's body movements in a virtual space requires extensive processing. Advances in computer graphics (CG) have made it relatively easy to create visually complex virtual environments, making technology for realizing corresponding auditory information important. Additionally, when processing from sound information to output sound information is performed in advance, a large memory area is required to store the pre-calculated processing results. Furthermore, transmitting such large amounts of processed data may require a wide communication bandwidth.

[0003] To achieve a more realistic sound environment, the number of objects that emit sound in the virtual three-dimensional space must be increased, and secondary sounds based on acoustic effects such as reflected sound, diffracted sound, and reverberation must be increased. Furthermore, these secondary sounds must be appropriately changed in response to the user's movements, which requires a large amount of processing power.

[0004] Japanese Patent Application Laid-Open No. 2020-18620

[0005] Therefore, an object of the present disclosure is to provide a sound processing device or the like that can generate an output sound signal appropriately in terms of the amount of processing.

[0006] A sound processing device according to one aspect of the present disclosure includes an acquisition unit that acquires sound information including an acoustic signal and information about the position of a sound source object within a three-dimensional sound field, a characteristic acquisition unit that acquires information about a user's hearing characteristics, and a reduction processing unit that, when generating an output sound signal from the acoustic signal included in the acquired sound information, reduces at least one sound signal based on the acquired information about the user's hearing characteristics, thereby generating the output sound signal that does not include the signal.

[0007] Furthermore, an acoustic processing method according to one aspect of the present disclosure is an acoustic processing method executed by a computer, and includes the steps of acquiring sound information including an acoustic signal and information about the position of a sound source object within a three-dimensional sound field, acquiring information about a user's hearing characteristics, and, when generating an output sound signal from the acoustic signal included in the acquired sound information, reducing at least one sound signal based on the acquired information about the user's hearing characteristics to generate the output sound signal that does not include that signal.

[0008] Furthermore, one aspect of the present disclosure can also be realized as a program for causing a computer to execute the above-described acoustic processing method.

[0009] These comprehensive or specific aspects may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0010] According to the present disclosure, it is possible to appropriately generate an output sound signal.

[0011] FIG. 1 is a schematic diagram showing a use example of a sound reproduction system according to an embodiment. FIG. 2 is a block diagram showing a functional configuration of the sound reproduction system according to an embodiment. FIG. 3 is a diagram for explaining an example of an audio signal according to an embodiment. FIG. 4 is a block diagram showing a functional configuration of an acquisition unit according to an embodiment. FIG. 5 is a block diagram showing a functional configuration of an output sound generation unit according to an embodiment. FIG. 6 is a diagram for explaining another example of a sound reproduction system according to an embodiment. FIG. 7 is a diagram for explaining another example of a sound reproduction system according to an embodiment. FIG. 8 is a diagram for explaining another example of a sound reproduction system according to an embodiment. FIG. 9 is a diagram for explaining another example of a sound reproduction system according to an embodiment. FIG. 10 is a diagram for explaining another example of a sound reproduction system according to an embodiment. FIG. 11 is a diagram for explaining another example of a sound reproduction system according to an embodiment. FIG. 12 is a diagram for explaining another example of a sound reproduction system according to an embodiment. FIG. 13 is a diagram for explaining another example of a sound reproduction system according to an embodiment. FIG. 14 is a diagram for explaining another example of a sound reproduction system according to an embodiment. FIG. 15 is a diagram for explaining a specific example of a sound reproduction system according to Example 1 of an embodiment. FIG. 16 is a diagram for explaining a specific example of a sound reproduction system according to Example 1 of an embodiment. FIG. 17 is a diagram for explaining a specific example of the sound reproduction system according to Example 1 of the embodiment. FIG. 18 is a diagram for explaining a specific example of the sound reproduction system according to Example 1 of the embodiment. FIG. 19 is a diagram for explaining a specific example of the sound reproduction system according to Example 1 of the embodiment. FIG. 20A is a diagram for explaining a specific example of the sound reproduction system according to Example 1 of the embodiment. FIG. 20B is a diagram for explaining a specific example of the sound reproduction system according to Example 1 of the embodiment. FIG. 21 is a diagram for explaining a specific example of the sound reproduction system according to Example 1 of the embodiment. FIG. 22 is a diagram for explaining a specific example of the sound reproduction system according to Example 1 of the embodiment. FIG. 23 is a diagram for explaining a specific example of the sound reproduction system according to Example 1 of the embodiment. FIG. 24 is a diagram for explaining a specific example of the sound reproduction system according to Example 1 of the embodiment.FIG. 25 is a diagram for explaining a specific example of the sound reproduction system according to Example 1 of the embodiment. FIG. 26 is a diagram for explaining a specific example of the sound reproduction system according to Example 1 of the embodiment. FIG. 27 is a diagram for explaining a specific example of the sound reproduction system according to Example 1 of the embodiment. FIG. 28 is a diagram for explaining a specific example of the sound reproduction system according to Example 1 of the embodiment. FIG. 29 is a diagram for explaining a specific example of the sound reproduction system according to Example 2 of the embodiment. FIG. 30 is a diagram for explaining a specific example of the sound reproduction system according to Example 2 of the embodiment. FIG. 31 is a diagram for explaining a specific example of the sound reproduction system according to Example 2 of the embodiment. FIG. 32 is a diagram for explaining a specific example of the sound reproduction system according to Example 2 of the embodiment. FIG. 33 is a diagram for explaining a specific example of the sound reproduction system according to Example 2 of the embodiment. FIG. 34 is a diagram for explaining a specific example of the sound reproduction system according to Example 2 of the embodiment. FIG. 35 is a diagram for explaining a specific example of the sound reproduction system according to Example 2 of the embodiment. FIG. 36 is a diagram for explaining a specific example of the sound reproduction system according to Example 2 of the embodiment. FIG. 37 is a diagram for explaining a specific example of the sound reproduction system according to Example 2 of the embodiment. FIG. 38 is a diagram for explaining a specific example of a sound reproduction system according to Example 2 of the embodiment. FIG. 39 is a diagram for explaining a specific example of a sound reproduction system according to Example 2 of the embodiment. FIG. 40 is a diagram for explaining a specific example of a sound reproduction system according to Example 2 of the embodiment. FIG. 41 is a diagram for explaining a specific example of a sound reproduction system according to Example 2 of the embodiment. FIG. 42 is a diagram for explaining a specific example of a sound reproduction system according to Example 2 of the embodiment. FIG. 43 is a diagram for explaining a specific example of a sound reproduction system according to Example 2 of the embodiment. FIG. 44 is a diagram for explaining a specific example of a sound reproduction system according to Example 2 of the embodiment. FIG. 45 is a diagram for explaining a specific example of a sound reproduction system according to Example 2 of the embodiment. FIG. 46 is a diagram for explaining a specific example of a sound reproduction system according to Example 2 of the embodiment. FIG. 47 is a diagram for explaining a specific example of a sound reproduction system according to Example 2 of the embodiment.FIG. 48 is a diagram for explaining a specific example of a sound reproduction system according to Example 2 of the embodiment. FIG. 49 is a diagram for explaining a specific example of a sound reproduction system according to Example 2 of the embodiment. FIG. 50 is a diagram for explaining a specific example of a sound reproduction system according to Example 2 of the embodiment. FIG. 51 is a diagram for explaining a specific example of a sound reproduction system according to Example 2 of the embodiment. FIG. 52 is a diagram for explaining a specific example of a sound reproduction system according to Example 2 of the embodiment. FIG. 53 is a diagram for explaining a specific example of a sound reproduction system according to Example 2 of the embodiment. FIG. 54 is a diagram for explaining a specific example of a sound reproduction system according to Example 2 of the embodiment. FIG. 55 is a diagram for explaining a specific example of a sound reproduction system according to Example 2 of the embodiment. FIG. 56 is a diagram for explaining a specific example of a sound reproduction system according to Example 2 of the embodiment. FIG. 57 is a diagram for explaining a specific example of a sound reproduction system according to a modified example of the embodiment. FIG. 58 is a diagram for explaining a specific example of a sound reproduction system according to a modified example of the embodiment. FIG. 59 is a diagram for explaining a specific example of a sound reproduction system according to a modified example of the embodiment. FIG. 60 is a diagram for explaining a specific example of a sound reproduction system according to a modified example of the embodiment. FIG. 61 is a diagram for explaining a specific example of a sound reproduction system according to a modified example of the embodiment. FIG. 62 is a diagram for explaining a specific example of a sound reproduction system according to a modified example of the embodiment. FIG. 63 is a diagram for explaining a specific example of a sound reproduction system according to a modified example of the embodiment. FIG. 64 is a diagram for explaining a specific example of a sound reproduction system according to a modified example of the embodiment. FIG. 65 is a diagram for explaining a specific example of a sound reproduction system according to a modified example of the embodiment. FIG. 66 is a diagram for explaining a specific example of a sound reproduction system according to a modified example of the embodiment. FIG. 67 is a diagram for explaining a specific example of a sound reproduction system according to a modified example of the embodiment. FIG. 68 is a diagram for explaining a specific example of a sound reproduction system according to a modified example of the embodiment.

[0012] (Knowledge that forms the basis of the disclosure) Conventionally, a technology related to sound reproduction that allows a user to perceive stereoscopic sound in a virtual three-dimensional space (hereinafter sometimes referred to as a three-dimensional sound field) has been known (see, for example, Patent Document 1). Using this technology, a user can perceive sound as if a sound source object exists at a predetermined position in the virtual space and the sound is coming from that direction. In order to localize a sound image at a predetermined position in the virtual three-dimensional space in this way, for example, calculation processing is required for a sound signal emitted by a sound source object (also referred to as a sound emitted from the sound source object or a reproduced sound) to generate a sound arrival time difference between the two ears and a sound level difference (or sound pressure difference) between the two ears that causes the sound to be perceived as stereoscopic sound. Such calculation processing is performed by applying a stereophonic filter. A stereophonic filter is an information processing filter that, when an output sound signal obtained by applying the filter to original sound information is reproduced, causes the position (such as the direction and distance) of the sound, the size of the sound source, the width of the space, and the like to be perceived with a three-dimensional effect.

[0013] As an example of the computational process for applying such a stereophonic filter, a process is known in which a head-related transfer function (HRTF) is convolved with a target sound signal to make the sound perceived as coming from a predetermined direction. By performing this HRTF convolution process at a sufficiently fine angle with respect to the direction of arrival of the reproduced sound from the position of the sound source object to the user's position, the sense of realism experienced by the user is improved.

[0014] In recent years, there has been active development of technologies related to virtual reality (VR). Virtual reality focuses on appropriately changing the position of a sound source object in a virtual three-dimensional space in response to a user's movements, allowing the user to experience the sensation of moving within the virtual space. To achieve this, it is necessary to move the localization position of a sound image in the virtual space relative to the user's movements. This processing has been performed by applying a stereophonic filter, such as the head-related transfer function described above, to the original sound information. However, when a user moves within a three-dimensional space, the sound transmission path changes from moment to moment due to changes in the positional relationship between the sound source object and the user, such as due to sound reverberation and interference. This requires determining the sound transmission path from the sound source object based on the positional relationship between the sound source object and the user, and convolving the transfer function to take into account sound reverberation and interference. However, such information processing requires a huge amount of processing, and an improvement in the sense of realism may not be achieved without a large-scale processing device.

[0015] Therefore, in order to reduce such an enormous amount of processing, attempts have been made to reduce a portion of the sounds being reproduced. Specifically, for each of the multiple sound source objects in a three-dimensional space, or for each of the multiple types of sounds generated from each of the sound source objects, rather than convolving a head-related transfer function with each of them, a portion of the sounds is reduced and then the head-related transfer function is convolved. By doing so, in the convolution of the head-related transfer function, which requires a particularly large amount of processing, that is, in the process of generating a stereophonic signal for output (in other words, an output signal or an output sound signal), the number of sound signals to be processed is reduced, and a significant reduction in the processing amount is expected.

[0016] However, since indiscriminate reduction of sound signals will result in sound degradation, the sound to be reduced is determined in consideration of the user's hearing characteristics in order to suppress this sound degradation. This makes it possible to realize a sound processing device that can suppress sound degradation while reducing the amount of processing.

[0017] A more specific outline of the present disclosure is as follows.

[0018] A sound processing device according to a first aspect of the present disclosure includes an acquisition unit that acquires sound information including a sound signal and information about the position of a sound source object within a three-dimensional sound field, a characteristic acquisition unit that acquires information about a user's hearing characteristics, and a reduction processing unit that, when generating an output sound signal from the sound signal included in the acquired sound information, reduces at least one sound signal based on the acquired information about the user's hearing characteristics, thereby generating an output sound signal that does not include the signal.

[0019] In other words, the sound processing device according to the first aspect is a sound processing device that generates multiple sounds that reach a user directly and / or indirectly from one or more sound sources, and reduces one or more of the multiple sounds based on characteristics related to the user's hearing (listening characteristics).

[0020] According to such a sound processing device, sound signals are reduced based on information about the user's hearing characteristics, and an output sound signal that does not include the reduced sound signals can be generated. In other words, since the sounds to be reduced can be determined appropriately based on the user's hearing characteristics, it is possible to generate an output sound signal appropriately in terms of sound degradation and processing amount.

[0021] Furthermore, a sound processing device according to a second aspect is the sound processing device according to the first aspect, wherein the information relating to the user's hearing characteristics is information relating to whether or not the user can distinguish between two or more sounds arriving toward the user.

[0022] That is, the sound processing device according to the second aspect is a sound processing device in which the hearing-related characteristics are based on the ability to distinguish between two or more sounds that reach the user.

[0023] With such a sound processing device, it is possible to determine the appropriate sounds to be reduced based on information regarding whether two or more sounds coming toward the user can be distinguished, making it possible to generate an output sound signal appropriately in terms of sound degradation and processing volume.

[0024] Furthermore, a sound processing device according to a third aspect is the sound processing device according to the second aspect, wherein the information relating to the user's hearing characteristics includes information relating to the angles of two or more sounds arriving toward the user, and the reduction processing unit reduces the signal of at least one of the two or more sounds based on the angle indicated by the information included in the information relating to the user's hearing characteristics.

[0025] That is, the sound processing device according to the third aspect is a sound processing device that reduces sounds based on the ability to distinguish angles of two or more sounds that reach the user as characteristics related to hearing.

[0026] With such an audio processing device, it is possible to determine the appropriate sound to be reduced based on the angle indicated by the information contained in the information regarding the user's hearing characteristics, making it possible to generate an output sound signal appropriately in terms of sound degradation and processing volume.

[0027] Furthermore, a sound processing device according to a fourth aspect is the sound processing device according to the second aspect, wherein the information relating to the user's hearing characteristics includes information relating to the distance difference between two or more sounds arriving toward the user, and the reduction processing unit reduces the signal of at least one of the two or more sounds based on the distance difference indicated by the information included in the information relating to the user's hearing characteristics.

[0028] That is, the sound processing device according to the fourth aspect is a sound processing device that reduces sounds based on two sounds that reach the user and the distance from the user.

[0029] With such an audio processing device, it is possible to determine the appropriate sound to be reduced based on the distance difference indicated by the information contained in the information regarding the user's hearing characteristics, making it possible to generate an output sound signal appropriately in terms of sound degradation and processing volume.

[0030] Furthermore, a sound processing device according to a fifth aspect is the sound processing device according to the second aspect, wherein the information relating to the user's hearing characteristics includes information relating to a level ratio of two or more sounds arriving toward the user, and the reduction processing unit reduces the signal of at least one of the two or more sounds based on the level ratio indicated by the information included in the information relating to the user's hearing characteristics.

[0031] That is, the sound processing device according to the fifth aspect is a sound processing device in which the hearing-related characteristics are based on the level ratio of two or more sounds that reach the user.

[0032] With such an audio processing device, it is possible to determine the appropriate sound to be reduced based on the level ratio indicated by the information contained in the information regarding the user's hearing characteristics, making it possible to generate an output sound signal appropriately in terms of sound degradation and processing volume.

[0033] Furthermore, a sound processing device according to a sixth aspect is the sound processing device according to the second aspect, wherein the information relating to the user's hearing characteristics includes information relating to a signal energy ratio of two or more sounds arriving toward the user, and the reduction processing unit reduces the signal of at least one of the two or more sounds based on the signal energy ratio indicated by the information included in the information relating to the user's hearing characteristics.

[0034] That is, the sound processing device according to the sixth aspect is a sound processing device in which the characteristics related to hearing are based on signal energy that utilizes the human hearing characteristics of two or more sounds that reach the user.

[0035] With such an audio processing device, it is possible to determine the appropriate sounds to be reduced based on the signal energy ratio indicated by the information contained in the information regarding the user's hearing characteristics, making it possible to generate an output sound signal appropriately in terms of sound degradation and processing volume.

[0036] Furthermore, a sound processing device according to a seventh aspect is the sound processing device according to the second aspect, wherein the information relating to the user's hearing characteristics includes information relating to the angle and level ratio of two or more sounds arriving toward the user, and the reduction processing unit reduces the signal of at least one of the two or more sounds based on the angle and level ratio indicated by the information included in the information relating to the user's hearing characteristics.

[0037] That is, the sound processing device according to the seventh aspect is a sound processing device in which the characteristics related to hearing are defined by both the directions and level ratios of two or more sounds that reach the user.

[0038] With such an audio processing device, it is possible to determine the appropriate sound to be reduced based on the angle and level ratio indicated by the information contained in the information regarding the user's hearing characteristics, making it possible to generate an output sound signal appropriately in terms of sound degradation and processing volume.

[0039] Furthermore, an audio processing device according to an eighth aspect is the audio processing device according to the second aspect, wherein the information relating to the user's hearing characteristics includes information relating to the angle and signal energy ratio of two or more sounds arriving toward the user, and the reduction processing unit reduces the signal of at least one of the two or more sounds based on the angle and signal energy ratio indicated by the information relating to the user's hearing characteristics.

[0040] In other words, the sound processing device of the eighth aspect is a sound processing device in which the characteristics related to hearing are determined by both the directions of two or more sounds reaching the user and signal energy that utilizes human hearing characteristics.

[0041] With such an audio processing device, it is possible to determine the appropriate sound to be reduced based on the angle and signal energy ratio indicated by the information contained in the information regarding the user's hearing characteristics, making it possible to generate an output sound signal appropriately in terms of sound degradation and processing volume.

[0042] Furthermore, a sound processing device according to a ninth aspect is a sound processing device according to any one of the first to eighth aspects, wherein the information relating to the user's hearing characteristics includes information relating to the sensitivity of each direction of sound coming towards the user, and the reduction processing unit preferentially reduces sounds from directions with lower sensitivity over sounds from directions with higher sensitivity, based on the sensitivity indicated by the information relating to the user's hearing characteristics.

[0043] That is, the sound processing device according to the ninth aspect is a sound processing device in which the auditory characteristic is such that the sensitivity varies depending on the direction from which sound is incident on the user.

[0044] With such an audio processing device, it is possible to determine the appropriate sounds to be reduced based on the level of sensitivity indicated by the information contained in the information regarding the user's hearing characteristics, making it possible to generate an output sound signal appropriately in terms of sound degradation and processing volume.

[0045] A sound processing device according to a tenth aspect is the sound processing device according to the ninth aspect, wherein the sensitivity is higher the closer to the front of the user, and lower the closer to the back of the user.

[0046] That is, the sound processing device according to the tenth aspect is a sound processing device in which the characteristics related to hearing are high in sensitivity in front of the user and decrease from the side to the rear.

[0047] With such a sound processing device, it is possible to determine the sound to be appropriately reduced based on the level of sensitivity, with higher sensitivity being shown the closer to the front of the user and lower sensitivity being shown the closer to the back of the user, so it is possible to generate an output sound signal appropriately in terms of sound degradation and processing volume.

[0048] Furthermore, a sound processing device according to an eleventh aspect is the sound processing device according to the ninth aspect, in which the high and low sensitivity includes the distribution of sensitivity in the vertical direction of the user 360° and the distribution of sensitivity in the horizontal direction of the user 360°.

[0049] That is, the sound processing device according to the eleventh aspect is a sound processing device in which characteristics related to hearing are expressed by a model that represents 360-degree directions up, down, left, right (horizontal and vertical).

[0050] With such a sound processing device, it is possible to determine the sound to be appropriately reduced based on the level of sensitivity, including the user's 360° vertical sensitivity distribution and the user's 360° horizontal sensitivity distribution, making it possible to generate an output sound signal appropriately in terms of sound degradation and processing volume.

[0051] A sound processing device according to a twelfth aspect is the sound processing device according to the eleventh aspect, in which the distribution of sensitivity in the horizontal direction of 360° is finer than the distribution of sensitivity in the vertical direction of 360°.

[0052] That is, the sound processing device according to the twelfth aspect is a sound processing device in which the auditory characteristic is more sensitive to changes in the horizontal direction than to changes in the vertical direction.

[0053] According to such a sound processing device, the distribution of sensitivity in the horizontal direction of 360° can be set more finely than the distribution of sensitivity in the vertical direction of 360°.

[0054] Furthermore, a sound processing device according to a thirteenth aspect is a sound processing device according to any one of the first to twelfth aspects, in which the reduction processing unit includes a culling unit that reduces at least one sound signal by discarding the signal of the at least one sound.

[0055] That is, the sound processing device according to the thirteenth aspect is a sound processing device that reduces one or more sounds by culling.

[0056] According to such a sound processing device, by discarding at least one sound signal, the signal of at least one sound is reduced, thereby making it possible to generate an output sound signal appropriately in terms of the amount of processing.

[0057] Furthermore, a sound processing device according to a fourteenth aspect is a sound processing device according to any one of the first to twelfth aspects, in which the reduction processing unit includes an integration unit that reduces the two sound signals by discarding at least two sound signals and supplementing the at least two sound signals with one virtual sound signal that is an integration of the two sound signals.

[0058] That is, the sound processing device according to the fourteenth aspect is a sound processing device that reduces one or more sounds by integrating two or more sounds.

[0059] With such an audio processing device, by discarding at least two sound signals and supplementing them with a single virtual sound signal that integrates the at least two sound signals, it is possible to generate an output sound signal appropriately in terms of sound degradation and processing volume.

[0060] Furthermore, a sound processing device according to a fifteenth aspect is a sound processing device according to any one of the first to twelfth aspects, in which the reduction processing unit includes a culling unit that discards at least one sound signal and reduces the at least one sound signal, and an integration unit that discards at least two sound signals and reduces the at least two sound signals by supplementing them with one virtual sound signal obtained by integrating the at least two sound signals.

[0061] In other words, the sound processing device according to the fifteenth aspect is a sound processing device that includes both a culling unit that reduces one or more sounds by culling, and an integration unit that reduces one or more sounds by integrating two or more sounds.

[0062] According to such a sound processing device, by discarding at least one sound signal, the signal of the at least one sound is reduced, and by discarding at least two sound signals and compensating for one virtual sound signal that is an integration of the at least two sound signals, it is possible to generate an output sound signal appropriately in terms of sound degradation and processing amount.

[0063] Furthermore, a sound processing device according to a sixteenth aspect is a sound processing device according to any one of the first to fifteenth aspects, in which the reduction processing unit reduces at least one sound signal based on the acquired information on the user's hearing characteristics and the type of sound.

[0064] That is, the sound processing device according to the sixteenth aspect is a sound processing device that controls the operation of sound reduction depending on the type of sound to be reduced.

[0065] With such a sound processing device, it is possible to determine the appropriate sounds to be reduced based on the acquired information about the user's hearing characteristics and the type of sound, making it possible to generate an output sound signal appropriately in terms of sound degradation and processing volume.

[0066] Furthermore, a sound processing device according to a seventeenth aspect is the sound processing device according to the fourteenth or fifteenth aspect, in which the integration unit discards at least two sound signals and generates a virtual sound signal by adding the at least two sound signals.

[0067] That is, the sound processing device according to the seventeenth aspect is a sound processing device that integrates one or more sounds by adding two or more sounds together.

[0068] With such a sound processing device, it is possible to appropriately generate an output sound signal in terms of sound degradation and processing volume by discarding at least two sound signals and supplementing the virtual sound signal generated by adding the at least two sound signals.

[0069] Furthermore, an audio processing device according to an 18th aspect is the audio processing device according to the 17th aspect, in which the integration unit discards at least two sound signals, adjusts at least one of the phase and energy of at least one of the at least two sound signals, and generates a virtual sound signal by adding the at least two sound signals after adjustment.

[0070] That is, the sound processing device according to the eighteenth aspect is a sound processing device that adds sounds by performing either phase adjustment or energy adjustment on at least one of two or more sounds.

[0071] With such a sound processing device, at least two sound signals are discarded, and the at least two sound signals are supplemented by performing either phase adjustment or energy adjustment on at least one of the sound signals and adding them together to generate a virtual sound signal, thereby making it possible to generate an output sound signal appropriately in terms of sound degradation and processing volume.

[0072] Furthermore, a sound processing device according to a 19th aspect is a sound processing device according to any one of the 1st to 18th aspects, in which the reduction processing unit gradually reduces the signal of at least one sound in the time domain.

[0073] In other words, the sound processing device of the 19th aspect is a sound processing device that includes processing that smoothly transitions (in the time domain) from before the change to after the change when the sound or the number of sounds to be integrated changes over time.

[0074] According to such a sound processing device, at least one sound signal is gradually reduced in the time domain, so that the sense of incongruity caused by the reduction of sound can be reduced.

[0075] Furthermore, a sound processing device according to a twentieth aspect is a sound processing device according to any one of the first to nineteenth aspects, in which the reduction processing unit performs at least one of discarding at least one sound signal input to at least one of the processes for generating each of a plurality of sound signals from an acoustic signal before the process, and discarding at least one sound signal generated in the process after the process for generating each of a plurality of sound signals from an acoustic signal.

[0076] In other words, the sound processing device of the twentieth aspect is a sound processing device in which the culling unit and the integration unit are arranged before or after a processing unit that generates sound that reaches the user directly and / or indirectly from the sound source.

[0077] With such a sound processing device, it is possible to discard at least one sound signal either before or after generating multiple sound signals, thereby generating an output sound signal appropriately in terms of processing volume.

[0078] Furthermore, a sound processing device according to a 21st aspect is a sound processing device according to any one of the 1st to 20th aspects, in which the reduction processing unit performs at least one of discarding at least one sound signal input to a process for generating each of a plurality of sound signals from an acoustic signal before the process for generating at least a diffracted sound, and discarding at least one diffracted sound signal generated in the process for generating each of a plurality of sound signals from an acoustic signal after the process for generating at least a diffracted sound.

[0079] That is, the sound processing device according to the twenty-first aspect is a sound processing device in which at least one of the culling unit and the integrating unit is arranged before or after the diffracted sound generating unit.

[0080] According to such a sound processing device, it is possible to discard at least one sound signal either before or after the generation of a diffracted sound signal, thereby generating an output sound signal appropriately in terms of the amount of processing.

[0081] An acoustic processing method executed by a computer, comprising the steps of: acquiring sound information including an acoustic signal and information about the position of a sound source object within a three-dimensional sound field; acquiring information about a user's hearing characteristics; and, when generating an output sound signal from the acoustic signal included in the acquired sound information, reducing at least one sound signal based on the acquired information about the user's hearing characteristics to generate an output sound signal that does not include the signal.Furthermore, an acoustic processing method according to a 22nd aspect is an acoustic processing method executed by a computer, comprising the steps of: acquiring sound information including an acoustic signal and information about the position of a sound source object within a three-dimensional sound field; acquiring information about the user's hearing characteristics; and, when generating an output sound signal from the acoustic signal included in the acquired sound information, reducing at least one sound signal based on the acquired information about the user's hearing characteristics to generate an output sound signal that does not include the signal.

[0082] This can provide the same effects as the sound processing device described above.

[0083] A program according to a twenty-third aspect is a program for causing a computer to execute the above-described acoustic processing method.

[0084] This makes it possible to achieve the same effects as the above-described acoustic processing method using a computer.

[0085] Furthermore, these comprehensive or specific aspects may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0086] Hereinafter, embodiments will be described in detail with reference to the drawings. Note that the embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not recited in independent claims will be described as optional components. Note that each figure is a schematic diagram and is not necessarily an exact illustration. Furthermore, in each figure, substantially identical components are assigned the same reference numerals, and duplicated descriptions may be omitted or simplified.

[0087] In the following description, elements may be assigned ordinal numbers such as first, second, and third. These ordinal numbers are assigned to elements in order to identify them and do not necessarily correspond to a meaningful order. These ordinal numbers may be rearranged, newly added, or removed as appropriate.

[0088] In the following description, an acoustic signal included in sound information may be described, but the acoustic signal may also be referred to as a voice signal or a sound signal. In other words, in the present disclosure, the acoustic signal has the same meaning as the voice signal or the sound signal.

[0089] (Embodiment) [Overview] First, an overview of an audio reproduction system according to an embodiment will be described. Fig. 1 is a schematic diagram showing an example of use of an audio reproduction system according to an embodiment. Fig. 1 shows a user 99 using an audio reproduction system 100.

[0090] The sound reproduction system 100 shown in FIG. 1 is used simultaneously with, for example, a three-dimensional video reproduction device 300. By simultaneously viewing three-dimensional images and three-dimensional sound, the image enhances the auditory sense of realism, and the sound enhances the visual sense of realism, allowing the user to experience the image and sound as if they were actually at the scene where they were captured. For example, when an image (moving image) of people having a conversation is displayed, it is known that even if the localization of the sound image (sound source object) of the conversation sound is not aligned with the person's mouth, the user 99 will perceive the conversation sound as coming from the person's mouth. In this way, the visual information can correct the position of the sound image, and the image and sound can be combined to enhance the sense of realism.

[0091] The three-dimensional video reproduction device 300 is an image display device worn on the head of the user 99. Therefore, the three-dimensional video reproduction device 300 moves integrally with the head of the user 99. For example, as shown in the figure, the three-dimensional video reproduction device 300 is a glasses-type device that is supported by the ears and nose of the user 99.

[0092] The three-dimensional video reproduction device 300 changes the displayed image in accordance with the movement of the user 99's head, thereby making the user 99 perceive the movement of his or her head in the three-dimensional image space. In other words, when an object in the three-dimensional image space is located in front of the user 99, if the user 99 turns to the right, the object moves to the left of the user 99, and if the user 99 turns to the left, the object moves to the right of the user 99. In this way, the three-dimensional video reproduction device 300 moves the three-dimensional image space in the opposite direction to the movement of the user 99.

[0093] The 3D video playback device 300 displays two images with a parallax difference to each of the user's 99's left and right eyes. The user 99 can perceive the three-dimensional position of an object on the image based on the parallax difference between the displayed images. Note that when the user 99 uses the audio playback system 100 with their eyes closed, for example, when using it to play healing sounds for sleep induction, the 3D video playback device 300 does not need to be used at the same time. In other words, the 3D video playback device 300 is not an essential component of the present disclosure. In addition to dedicated video display devices, the 3D video playback device 300 may also be a general-purpose mobile terminal owned by the user 99, such as a smartphone or tablet device.

[0094] Such general-purpose mobile terminals are equipped with not only a display for displaying images but also various sensors for detecting the terminal's posture and movement. Furthermore, they are also equipped with a processor for information processing, and are capable of connecting to a network to transmit and receive information to and from a server device such as a cloud server. In other words, the 3D video playback device 300 and the audio playback system 100 can be realized by combining a smartphone with general-purpose headphones or the like that do not have an information processing function.

[0095] As in this example, the head movement detection function, the video presentation function, the video information processing function for presentation, the sound presentation function, and the sound information processing function for presentation may be appropriately arranged in one or more devices to realize the 3D video reproduction device 300 and the sound reproduction system 100. If the 3D video reproduction device 300 is not required, it is sufficient to appropriately arrange the head movement detection function, the sound presentation function, and the sound information processing function for presentation in one or more devices. For example, the sound reproduction system 100 can be realized by a processing device such as a computer or smartphone having a sound information processing function for presentation, and headphones or the like having a head movement detection function and a sound presentation function.

[0096] The sound reproduction system 100 is a sound presentation device that is worn on the head of the user 99. Therefore, the sound reproduction system 100 moves integrally with the head of the user 99. For example, the sound reproduction system 100 in this embodiment is a so-called over-ear headphone type device. Note that there are no particular limitations on the form of the sound reproduction system 100, and it may be, for example, two earplug-type devices that are worn independently on the left and right ears of the user 99.

[0097] The sound reproduction system 100 changes the sound presented in accordance with the movement of the head of the user 99, thereby making the user 99 perceive as if he or she is moving his or her head within the three-dimensional sound field. For this reason, as described above, the sound reproduction system 100 moves the three-dimensional sound field in the opposite direction to the movement of the user 99.

[0098] Here, when the user 99 moves within the three-dimensional sound field, the position of the sound source object changes relative to the user 99's position within the three-dimensional sound field. As a result, each time the user 99 moves, it is necessary to perform calculation processing based on the positions of the sound source object and the user 99 to generate an output sound signal for playback. Because such processing typically requires a huge amount of processing, the present disclosure generates and outputs an output sound signal in which multiple sound signals constituting the output sound signal used for head-related transfer function convolution are reduced, from the perspective of reducing the amount of processing. As a result, the number of sound signals used for head-related transfer function convolution is reduced, and a significant reduction in processing volume is expected. In this case, if sound signals are selected indiscriminately as sound signals to be reduced, the user will perceive a deterioration in sound quality. Therefore, the present disclosure selects sound signals that correspond to the user's hearing characteristics as sound signals to be reduced. In other words, by selectively selecting sounds that have a relatively small impact on sound quality as sound signals to be reduced, taking the user's hearing characteristics into consideration, it is possible to reduce the amount of processing while preventing unnecessary deterioration in sound quality.

[0099] [Configuration] Next, the configuration of the sound reproduction system 100 according to this embodiment will be described with reference to Fig. 2. Fig. 2 is a block diagram showing the functional configuration of the sound reproduction system according to this embodiment.

[0100] As shown in FIG. 2, the sound reproduction system 100 according to this embodiment includes an information processing device 101, a communication module 102, a detector 103, a driver 104, and a database 105.

[0101] The information processing device 101 is an example of an audio processing device, and is an arithmetic device for performing various signal processing in the audio reproduction system 100. The information processing device 101 includes a processor and a memory, such as a computer, and is realized by the processor executing a program stored in the memory. Execution of this program provides functions related to each functional unit described below.

[0102] The information processing device 101 includes an acquisition unit 111, a path calculation unit 121, an output sound generation unit 131, and a signal output unit 141. Details of each functional unit included in the information processing device 101 will be described below together with details of the configuration other than the information processing device 101.

[0103] The communication module 102 is an interface device for accepting input of sound information to the sound reproduction system 100. The communication module 102 includes, for example, an antenna and a signal converter, and receives sound information from an external device via wireless communication. More specifically, the communication module 102 receives, using the antenna, a wireless signal indicating sound information converted into a format for wireless communication, and reconverts the wireless signal into sound information using the signal converter. In this way, the sound reproduction system 100 acquires sound information from the external device via wireless communication. The sound information acquired by the communication module 102 is acquired by the acquisition unit 111. In this way, the acquisition unit 111 is an example of a sound acquisition unit. The sound information is input to the information processing device 101 in the above manner. Note that communication between the sound reproduction system 100 and the external device may be performed via wired communication.

[0104] The sound information acquired by the sound reproduction system 100 is encoded in a predetermined format, such as MPEG-H 3D Audio (ISO / IEC 23008-3). As an example, the encoded sound information includes information about the sound reproduced by the sound reproduction system 100 and information about the localization position when the sound image of the sound is localized at a predetermined position in a three-dimensional sound field (i.e., perceived as sound coming from a predetermined direction). The sound information can also be interpreted as information about a sound source object. In other words, the sound information includes the position of the sound source object in the three-dimensional sound field and the sound produced by the sound source object.

[0105] As described above, sound information is obtained as input data, and includes an audio signal (acoustic signal), which is information about the reproduced sound, and other information, such as information about the position of a sound source object in a three-dimensional sound field. The other information may also include information for defining a three-dimensional sound field. Therefore, the other information may be collectively referred to as information about space (spatial information), including information about the position of a sound source object and information for defining a three-dimensional sound field. When the audio signal is viewed as the main focus, the input data can be said to be sound information in which other information (metadata) is attached to the audio signal. When the spatial information is viewed as the main focus, the input data can be said to be information in which the audio signal is attached to the spatial information. Alternatively, since the input data has both aspects, the input data can also be considered as sound spatial information.

[0106] As a specific example, the sound information includes information about a plurality of sounds including a first reproduced sound and a second reproduced sound, and the sound images of the respective sounds when reproduced are localized so that they are perceived as coming from different positions in a three-dimensional sound field. Therefore, the sound source object of the first reproduced sound is localized at a first position in the three-dimensional sound field, and the sound source object of the second reproduced sound is localized at a second position in the three-dimensional sound field. Thus, the sound information may include a plurality of sounds. In other words, the sound information may include a plurality of audio signals corresponding to the first reproduced sound and the second reproduced sound, respectively, and the positions of a plurality of sound source objects at first and second positions that correspond one-to-one to the plurality of audio signals.

[0107] 3 is a diagram illustrating an example of an audio signal according to an embodiment. For example, as shown in (a) of FIG. 3, the sound information may include an audio signal of a first direct sound arriving from a first position (from a first direction) to the position of the user 99 and an audio signal of a second direct sound arriving from a second position (from a second direction) to the position of the user 99. The sound information immediately after acquisition may include only information about the reproduced sound. In this case, information about a predetermined position may be acquired separately, and the subsequent processing may be performed once the information is collected. As described above, the sound information includes first sound information about the first reproduced sound and second sound information about the second reproduced sound. However, multiple pieces of sound information each including these may be acquired separately and played back simultaneously (i.e., treated as a single piece of sound information), thereby localizing sound images at different positions in a three-dimensional sound field and causing the reproduced sounds to arrive from different directions.

[0108] Alternatively, the sound information may include multiple audio signals and the position of a single sound source object that corresponds to the multiple audio signals in a many-to-one relationship. For example, such sound information is used in a situation where multiple sounds are reproduced from a sound source object. For example, each of the multiple audio signals corresponds to a direct sound that arrives directly from the position of the sound source object to the position of the user 99, and a secondary sound (sound generated by indirect propagation) that accompanies the direct sound and arrives via a path different from that of the direct sound.

[0109] For example, as shown in (b) of FIG. 3 , the sound information immediately after acquisition includes an audio signal related to the direct sound, and is converted into sound information including audio signals such as reverberation, primary reflection, and diffraction through a conversion process for calculating secondary sounds. This conversion process for calculating secondary sounds uses information on the spatial environment conditions of the three-dimensional sound field (e.g., the position, reflection, and diffraction characteristics of objects in the three-dimensional sound field). Thus, secondary sounds are computationally generated from sound information related to a single playback sound based on the spatial environment conditions of the three-dimensional sound field. Therefore, the secondary sounds are not included in the sound information immediately after acquisition, and sound information including these secondary sounds is generated through the conversion process for calculating the secondary sounds. Another secondary sound may be generated from one secondary sound through its propagation. Note that the information on the spatial environment conditions is part of the spatial information and may be acquired together with the audio signal from the input sound information. Alternatively, the audio signal and the spatial information may be acquired separately. That is, the sound information may be acquired from a single file or bitstream, or may be acquired separately by dividing it into multiple files or bitstreams. For example, the audio signal and the spatial information may be acquired from separate files or bitstreams, or the audio signal and the spatial information may each be acquired from a plurality of files or bitstreams.

[0110] The example in Figure 3(b) illustrates that a secondary reflected sound is generated from a primary reflected sound. As shown in the figure, these secondary sounds are assigned tags that identify their relationships with each other, such as parent, child, and grandchild, as information regarding the genealogy of their generation relationships from the direct sound (in other words, the generation system). Alternatively, when the direct sound is defined as the 0th generation, the secondary sounds may be quantified as the number of generations, such as the first generation to which the first reflected sound belongs and the second generation to which the second reflected sound belongs. Note that the number of generations that are allowed to be generated from the direct sound may be set according to the scale of the computational resources.

[0111] As described above, there is no particular limitation on the form of the input sound information, and the sound reproduction system 100 may be provided with an acquisition unit 111 that can accommodate various forms of sound information.

[0112] An example of the acquisition unit 111 will now be described with reference to Fig. 4. Fig. 4 is a block diagram showing the functional configuration of the acquisition unit according to the embodiment. As shown in Fig. 4, the acquisition unit 111 in this embodiment includes, for example, an encoded sound information input unit 112, a decoding processing unit 113, and a sensing information input unit 114.

[0113] The encoded sound information input unit 112 is a processing unit to which the coded (in other words, encoded) sound information acquired by the acquisition unit 111 is input. The encoded sound information input unit 112 outputs the input sound information to the decoding processing unit 113. The decoding processing unit 113 is a processing unit that decodes (in other words, decodes) the sound information output from the encoded sound information input unit 112 to generate a playback sound and the position of a sound source object included in the sound information in a format that is used for subsequent processing. The sensing information input unit 114 will be described below together with the function of the detector 103.

[0114] The detector 103 is a device for detecting the speed of movement of the head of the user 99. The detector 103 is configured by combining various sensors used for detecting movement, such as a gyro sensor and an acceleration sensor. In this embodiment, the detector 103 is built into the sound reproduction system 100. However, the detector 103 may be built into an external device, such as a three-dimensional image reproduction device 300 that operates in response to the movement of the head of the user 99 in the same way as the sound reproduction system 100. In this case, the detector 103 does not need to be included in the sound reproduction system 100. Alternatively, the detector 103 may be an external imaging device or the like that captures the movement of the head of the user 99 and detects the movement of the user 99 by processing the captured image.

[0115] The detector 103 is, for example, fixed integrally to the housing of the sound reproduction system 100 and detects the speed of movement of the housing. After the sound reproduction system 100 including the housing is worn by the user 99, it moves integrally with the head of the user 99, and as a result, the detector 103 can detect the speed of movement of the head of the user 99.

[0116] The detector 103 may detect, for example, the amount of head movement of the user 99 as the amount of rotation about at least one of three axes that are orthogonal to each other in three-dimensional space as the rotation axis, or may detect the amount of displacement about at least one of the three axes as the displacement direction. Furthermore, the detector 103 may detect both the amount of rotation and the amount of displacement as the amount of head movement of the user 99.

[0117] The sensing information input unit 114 acquires the movement speed of the user 99's head from the detector 103. More specifically, the sensing information input unit 114 acquires the amount of head movement of the user 99 detected by the detector 103 per unit time as the movement speed. In this way, the sensing information input unit 114 acquires at least one of the rotation speed and the displacement speed from the detector 103. The amount of head movement of the user 99 acquired here is used to determine the position and posture (in other words, coordinates and orientation) of the user 99 in the three-dimensional sound field. Therefore, the acquisition unit 111 also functions as a position acquisition unit via the sensing information input unit 114. In the sound reproduction system 100, the relative position of the sound image object with respect to the user 99 is determined based on the determined coordinates and orientation of the user 99, and sound is reproduced. Specifically, the above functions are realized by the path calculation unit 121 and the output sound generation unit 131.

[0118] The path calculation unit 121 includes an arrival direction calculation function that calculates the relative arrival direction of the reproduced sound from the position of the sound source object to the position of the user 99 based on the determined coordinates and orientation of the user 99, and the conversion process that calculates the secondary sound described above. Therefore, the path calculation unit 121 includes a function that calculates a propagation path from the sound source object and calculates the secondary sound and the arrival direction of the secondary sound that arrives at the position of the user 99 through indirect propagation of the reproduced sound according to the calculated propagation path of the reproduced sound. The arrival direction of the secondary sound includes additional information, such as what object the sound is reflected from in the case of a reflected sound and the attenuation rate at the time of reflection. The additional information is included in the arrival direction of the secondary sound calculated from the input sound information. In other words, the additional information is computationally generated and acquired from the sound information.

[0119] To summarize the spatial information, the spatial information includes further information such as the spatial position of a sound source object in a space (three-dimensional sound field) (information on the position of the sound source object), sound reflection and diffraction characteristics at the sound source object (together with information on the conditions of the spatial environment), and the size of the three-dimensional sound field. Based on the spatial information, the path calculation unit 121 generates secondary sounds depending on which sound source object the reproduced sound is reflected or diffracted by, and calculates, as additional information, the direction from which the secondary sounds arrive and the volume of the secondary sounds after attenuation by reflection or diffraction. The sound information (input data) includes spatial information in the form of audio signals and accompanying metadata. As described above, the spatial information includes, as information other than the audio signal, information necessary to turn the sound into stereophonic sound and position the sound source object in a three-dimensional sound field, and / or information used to calculate information necessary to turn the sound into stereophonic sound and position the sound source object in a three-dimensional sound field.

[0120] The path calculation unit 121 may be realized by any processing as long as it can calculate the arrival direction of the reproduced sound when the reproduced sound reaches the user as a direct sound and can calculate the arrival direction of a secondary sound that arrives at the position of the user 99 due to secondary propagation of the reproduced sound. The path calculation unit 121 determines from which direction in the three-dimensional sound field the reproduced sound and the secondary sound are to be perceived by the user 99 as coming from, based on the coordinates and orientation of the user 99, and processes the sound information so that the output sound signal is perceived as such a sound when it is reproduced.

[0121] The output sound generating unit 131 is a processing unit that processes information about the reproduced sound included in the sound information to generate an output sound signal.

[0122] An example of the output sound generation unit 131 will now be described with reference to Fig. 5. Fig. 5 is a block diagram showing the functional configuration of the output sound generation unit according to an embodiment. As shown in Fig. 5, the output sound generation unit 131 in this embodiment includes, for example, a reduction processing unit 132, which includes a culling unit 133 and an integration unit 134.

[0123] The reduction processing unit 132 is a processing unit that processes sound information by the path calculation unit 121, etc., and determines from multiple sound signals, such as direct sound, reverberant sound, (primary, secondary and higher-order) reflected sound, and indirect sound such as diffracted sound, sound signals that are unlikely to cause an auditory difference even if reduced, i.e., sound signals that are unlikely to cause the user 99 to perceive a deterioration in sound, and reduces those signals.

[0124] The reduction processing unit 132 uses the culling unit 133 to stop the generation of a sound signal, or to perform a culling process that discards the generated sound signal, so that the sound signal is not included in a subsequent output sound signal. The culling unit 133 is a processing unit that discards the specific sound signal determined as the sound to be reduced in this way. Note that discarding here is in a broad sense, including discarding a signal by stopping the generation itself.

[0125] Furthermore, the reduction processing unit 132 discards two or more sound signals using the integrating unit 134, and instead integrates the discarded sound signals into a smaller number of virtual sounds to generate one or more virtual sounds that virtually replace the two or more sounds, so that the two or more sound signals are not included in the subsequent output sound signal, and fewer virtual sound signals are included in the subsequent output sound signal. The integrating unit 134 is a processing unit that discards two or more specific sound signals determined as sounds to be reduced in this way, and generates a smaller number of virtual sound signals to replace them.

[0126] Here, the reduction processing unit 132 determines specific sounds to be reduced based on the user's hearing characteristics. The user's hearing characteristics indicate, for example, the relationship between the ease of distinguishing sounds, including whether the user can distinguish two or more sounds from each other, and the difference in physical characteristics between the two or more sounds. In other words, if the difference in physical characteristics between two or more sounds in a certain combination is relatively easy to distinguish, the reduction processing unit 132 does not reduce these sounds because the sound quality is likely to be degraded even if any of the sounds is reduced. Instead, if the difference in physical characteristics between two or more sounds in another combination is relatively difficult to distinguish, the reduction processing unit 132 reduces at least one of the sounds because the sound quality is likely to be degraded even if any of the sounds is reduced. The process of determining sounds to be reduced based on the user's hearing characteristics will be described in more detail in the examples described below.

[0127] Referring again to FIG. 2 , the output sound generation unit 131 acquires a head-related transfer function used to generate an output sound signal from the database 105. The database 105 is an information storage device that functions both as a storage device for storing information and as a storage controller that reads out the stored information and outputs it to an external component. The database 105 stores a head-related transfer function for each direction of arrival of the sound from the user 99. The head-related transfer functions included in the database 105 are a set of general-purpose head-related transfer functions that can be used by everyone, a set of head-related transfer functions optimized for each individual user 99, or a set of head-related transfer functions that are publicly available. The database 105 receives an inquiry from the output sound generation unit 131 using the direction of arrival as a query, and outputs a head-related transfer function corresponding to the direction of arrival to the output sound generation unit 131. In addition, the output sound generation unit 131 may output the entire set of head-related transfer functions, or may output the characteristics of the set of head-related transfer functions itself.

[0128] The signal output unit 141 is a functional unit that outputs the generated output sound signal to the driver 104. The signal output unit 141 generates a waveform signal by performing signal conversion from a digital signal to an analog signal based on the output sound signal, and then causes the driver 104 to generate sound waves based on the waveform signal, thereby presenting sound to the user 99. The driver 104 includes, for example, a diaphragm and a drive mechanism such as a magnet and a voice coil. The driver 104 operates the drive mechanism in response to the waveform signal, causing the drive mechanism to vibrate the diaphragm. In this way, the driver 104 generates sound waves by vibrating the diaphragm in response to the output sound signal (this means "reproducing" the output sound signal; in other words, "reproducing" does not include perception by the user 99). The sound waves propagate through the air and reach the ears of the user 99, and the user 99 perceives the sound.

[0129] [Another Configuration Example] In the above example, the sound reproduction system 100 according to the present embodiment is an audio presentation device, and has been described as including an information processing device 101, a communication module 102, a detector 103, a database 105, and a driver 104. However, the functions of the sound reproduction system 100 may be realized by a plurality of devices or by a single device. Specific examples will be described using Figures 6 to 14. Figures 6 to 14 are diagrams for explaining other examples of the sound reproduction system according to the embodiment.

[0130] For example, the information processing device 601 may be included in the audio presentation device 602, and the audio presentation device 602 may perform both acoustic processing and sound presentation. Alternatively, the information processing device 601 and the audio presentation device 602 may share the acoustic processing described in the present disclosure, or a server connected to the information processing device 601 or the audio presentation device 602 via a network may perform part or all of the acoustic processing described in the present disclosure.

[0131] In the above explanation, the information processing device 601 is referred to as such, but when the information processing device 601 performs acoustic processing by decoding a bitstream generated by encoding at least a portion of the data of an audio signal or spatial information used for acoustic processing, the information processing device 601 may be referred to as a decoding device, and the acoustic reproduction system 100 (i.e., the stereophonic sound reproduction system 600 in the figure) may be referred to as a decoding processing system.

[0132] Here, an example will be described in which the sound reproduction system 100 functions as a decoding processing system.

[0133] <Example of Encoding Device> FIG. 7 is a functional block diagram showing the configuration of an encoding device 700 that is an example of an encoding device according to the present disclosure.

[0134] Input data 701 is data to be coded, including spatial information and / or an audio signal, that is input to an encoder 702. Details of the spatial information will be described later.

[0135] The encoder 702 encodes the input data 701 to generate encoded data 703. The encoded data 703 is, for example, a bit stream generated by the encoding process.

[0136] The memory 704 stores the encoded data 703. The memory 704 may be, for example, a hard disk or a solid-state drive (SSD), or may be any other storage device.

[0137] In the above description, a bitstream generated by an encoding process is given as an example of the encoded data 703 stored in memory 704. However, data other than a bitstream may also be used. For example, the encoding device 700 may convert a bitstream into a predetermined data format and store the converted data in memory 704. The converted data may be, for example, a file or multiplexed stream storing one or more bitstreams. Here, the file may have a file format such as ISOBMFF (ISO Base Media File Format). The encoded data 703 may also be in the form of multiple packets generated by dividing the bitstream or file. When converting the bitstream generated by the encoder 702 into data other than the bitstream, the encoding device 700 may include a conversion unit (not shown), or the conversion process may be performed by a CPU (Central Processing Unit).

[0138] <Example of Decoding Device> FIG. 8 is a functional block diagram showing the configuration of a decoding device 800 that is an example of a decoding device according to the present disclosure.

[0139] The memory 804 stores, for example, the same data as the coded data 703 generated by the coding device 700. The memory 804 reads out the stored data and inputs it as input data 803 to the decoder 802. The input data 803 is, for example, a bitstream to be decoded. The memory 804 may be, for example, a hard disk or an SSD, or may be another storage device.

[0140] Note that the decoding device 800 may not use the data stored in the memory 804 as input data 803 as is, but may convert the read data and generate converted data as input data 803. The data before conversion may be, for example, multiplexed data storing one or more bitstreams. Here, the multiplexed data may be, for example, a file having a file format such as ISOBMFF. The data before conversion may also be in the form of multiple packets generated by dividing the bitstream or file. When converting data different from the bitstream read from the memory 804 into a bitstream, the decoding device 800 may be provided with a conversion unit (not shown), or the conversion process may be performed by a CPU.

[0141] Decoder 802 decodes input data 803 to generate audio signal 801 that is presented to a listener.

[0142] <Another Example of Encoding Device> Fig. 9 is a functional block diagram showing the configuration of an encoding device 900, which is another example of an encoding device of the present disclosure. In Fig. 9, components having the same functions as those in Fig. 7 are assigned the same reference numerals as those in Fig. 7, and descriptions of these components will be omitted.

[0143] The coding device 900 differs from the coding device 700 in that the coding device 700 includes a memory 704 for storing coded data 703, whereas the coding device 900 includes a transmitting unit 901 for transmitting coded data 703 to the outside.

[0144] The transmitter 901 transmits a transmission signal 902 to another device or a server based on the encoded data 703 or data in another data format generated by converting the encoded data 703. The data used to generate the transmission signal 902 is, for example, the bit stream, multiplexed data, file, or packet described in the encoding device 700.

[0145] <Another Example of Decoding Device> Fig. 10 is a functional block diagram showing the configuration of a decoding device 1000, which is another example of a decoding device of the present disclosure. In Fig. 10, components having the same functions as those in Fig. 8 are assigned the same reference numerals as those in Fig. 8, and descriptions of these components will be omitted.

[0146] The decoding device 800 differs from the decoding device 800 in that the decoding device 800 includes a memory 804 for reading out input data 803, whereas the decoding device 1000 includes a receiving unit 1001 for receiving input data 803 from the outside.

[0147] The receiving unit 1001 receives a received signal 1002, acquires received data, and outputs input data 803 to be input to the decoder 802. The received data may be the same as the input data 803 to be input to the decoder 802, or may be data in a data format different from that of the input data 803. If the received data is data in a data format different from that of the input data 803, the receiving unit 1001 may convert the received data into the input data 803, or a conversion unit or CPU (not shown) included in the decoding device 1000 may convert the received data into the input data 803. The received data is, for example, a bit stream, multiplexed data, a file, or a packet, as described in the encoding device 900.

[0148] <Functional Description of Decoder> FIG. 11 is a functional block diagram showing the configuration of a decoder 1100, which is an example of the decoder 802 in FIG. 8 or FIG.

[0149] The input data 803 is an encoded bitstream, and includes encoded audio data, which is an encoded audio signal, and metadata used for acoustic processing.

[0150] The spatial information management unit 1101 acquires metadata included in the input data 803 and analyzes the metadata. The metadata includes information describing elements that act on sounds arranged in a sound space. The spatial information management unit 1101 manages spatial information necessary for acoustic processing obtained by analyzing the metadata and provides the spatial information to the rendering unit 1103. Note that, although the information used for acoustic processing is referred to as spatial information in this disclosure, it may be called by other names. The information used for acoustic processing may be called, for example, sound space information or scene information. Furthermore, when the information used for acoustic processing changes over time, the spatial information input to the rendering unit 1103 may be called a space state, a sound space state, a scene state, or the like.

[0151] Furthermore, spatial information may be managed for each sound space or for each scene. For example, when different rooms are represented as virtual spaces, each room may be managed as a scene of a different sound space, or even if the same space is represented, spatial information may be managed as different scenes depending on the scene being represented. In managing spatial information, an identifier for identifying each piece of spatial information may be assigned. The spatial information data may be included in a bitstream, which is one form of input data 803, or the bitstream may include an identifier for the spatial information, and the spatial information data may be acquired from a source other than the bitstream. When the bitstream includes only the identifier for the spatial information, the identifier for the spatial information may be used during rendering to acquire the spatial information data stored in the memory of the acoustic signal processing device or an external server as input data.

[0152] Note that the information managed by the spatial information management unit 1101 is not limited to information included in the bitstream. For example, the input data 803 may include data indicating the characteristics or structure of a space acquired from a software application or server providing VR or AR, as data not included in the bitstream. Furthermore, for example, the input data 803 may include data indicating the characteristics or position of a listener or object, as data not included in the bitstream. Furthermore, the input data 803 may include, as information indicating the position of the listener, information acquired by a sensor provided in a terminal including a decoding device, or information indicating the position of the terminal estimated based on information acquired by the sensor. In other words, the spatial information management unit 1101 may communicate with an external system or server to acquire spatial information and the position of the listener. Furthermore, the spatial information management unit 1101 may acquire clock synchronization information from an external system and execute a process of synchronizing with the clock of the rendering unit 1103. Note that the space in the above description may be a virtually formed space, i.e., a VR space, or may be a real space or a virtual space corresponding to a real space, i.e., an AR space or an MR (Mixed Reality) space. The virtual space may also be called a sound field or a sound space. Furthermore, the information indicating a position in the above description may be information such as coordinate values ​​indicating a position within a space, information indicating a relative position with respect to a predetermined reference position, or information indicating the movement or acceleration of a position within a space.

[0153] The audio data decoder 1102 decodes the encoded audio data included in the input data 803 to obtain an audio signal.

[0154] The encoded audio data acquired by the stereophonic sound reproduction system 600 is a bitstream encoded in a predetermined format such as MPEG-H 3D Audio (ISO / IEC 23008-3). Note that MPEG-H 3D Audio is merely one example of an encoding method that can be used to generate the encoded audio data included in the bitstream, and the encoded audio data may be included in a bitstream encoded in another encoding method. For example, the encoding method used may be a lossy codec such as MP3 (MPEG-1 Audio Layer-3), AAC (Advanced Audio Coding), WMA (Windows Media Audio), AC3 (Audio Codec-3), or Vorbis, or a lossless codec such as ALAC (Apple Lossless Audio Codec) or FLAC (Free Lossless Audio Codec), or any other encoding method may be used. For example, PCM (Pulse Code Modulation) data may be a type of encoded audio data. In this case, the decoding process may be, for example, a process of converting an N-bit binary number into a number format (e.g., floating-point format) that can be processed by the rendering unit 1103, where the number of quantization bits of the PCM data is N.

[0155] The rendering unit 1103 receives an audio signal and spatial information as input, performs acoustic processing on the audio signal using the spatial information, and outputs an audio signal 801 after acoustic processing.

[0156] Before starting rendering, the spatial information management unit 1101 reads metadata of the input signal, detects rendering items such as objects or sounds defined in the spatial information, and transmits the detected items to the rendering unit 1103. After starting rendering, the spatial information management unit 1101 grasps changes over time in the spatial information and the position of the listener, and updates and manages the spatial information. The spatial information management unit 1101 then transmits the updated spatial information to the rendering unit 1103. The rendering unit 1103 generates and outputs an audio signal to which acoustic processing has been applied based on the audio signal included in the input data and the spatial information received from the spatial information management unit 1101.

[0157] The spatial information update process and the audio signal output process with added acoustic processing may be executed in the same thread, or the spatial information management unit 1101 and the rendering unit 1103 may be allocated to independent threads. When the spatial information update process and the audio signal output process with added acoustic processing are executed in different threads, the thread startup frequency may be set individually, or the processes may be executed in parallel.

[0158] By having the spatial information management unit 1101 and the rendering unit 1103 execute their processes in different, independent threads, computational resources can be preferentially allocated to the rendering unit 1103. Therefore, in the case of sound output processing in which even a slight delay cannot be tolerated, for example, sound output processing in which a delay of even one sample (0.02 msec) would cause a popping noise, can be safely performed. In this case, the allocation of computational resources to the spatial information management unit 1101 is limited. However, compared to audio signal output processing, updating spatial information is a less frequent process (e.g., a process such as updating the listener's facial orientation). Therefore, unlike audio signal output processing, updating spatial information does not necessarily require an instantaneous response, and therefore limiting the allocation of computational resources does not significantly affect the acoustic quality provided to the listener.

[0159] The space information may be updated periodically at preset times or intervals, or when preset conditions are met. The space information may be updated manually by a listener or a sound space manager, or may be triggered by a change in an external system. For example, when a listener operates a controller to instantly warp the position of their avatar, instantly advance or rewind the time, or when a virtual space manager suddenly changes the environment of the space, the thread in which the space information management unit 1101 is located may be started as a one-off interrupt process in addition to being started periodically.

[0160] The role of the information update thread that executes the spatial information update process is, for example, to update the position or orientation of the listener's avatar placed in the virtual space based on the position or orientation of the VR goggles worn by the listener, and to update the position of objects moving in the virtual space. These tasks are handled within a processing thread that runs relatively infrequently, on the order of several tens of Hz. Processing to reflect the properties of direct sound may be performed in such an infrequently occurring processing thread. This is because the properties of direct sound change less frequently than the frequency of audio processing frames for audio output. Doing so can relatively reduce the computational load of the process and also avoid the risk of pulsive noise occurring when information is updated at an unnecessarily fast frequency.

[0161] FIG. 12 is a functional block diagram showing the configuration of a decoder 1200, which is another example of the decoder 802 in FIG. 8 or 10.

[0162] Figure 12 differs from Figure 11 in that the input data 803 includes an unencoded audio signal rather than encoded audio data. The input data 803 includes a bitstream including metadata and an audio signal.

[0163] The spatial information management unit 1201 is the same as the spatial information management unit 1101 in FIG. 11, and therefore a description thereof will be omitted.

[0164] The rendering unit 1202 is the same as the rendering unit 1103 in FIG. 11, and therefore a description thereof will be omitted.

[0165] In the above description, the configuration in Fig. 12 is called a decoder, but it may also be called an audio processing unit that performs audio processing. Furthermore, a device including an audio processing unit may also be called an audio processing device rather than a decoding device. Furthermore, the audio signal processing device (information processing device 601) may also be called an audio processing device.

[0166] <Physical Configuration of Encoding Apparatus> Fig. 13 is a diagram showing an example of the physical configuration of an encoding apparatus. The encoding apparatus shown in Fig. 13 is an example of the encoding apparatuses 700 and 900 described above.

[0167] The encoding device of FIG. 13 includes a processor, a memory, and a communication IF.

[0168] The processor may be, for example, a CPU (Central Processing Unit), a DSP (Digital Signal Processor), or a GPU (Graphics Processing Unit), and the CPU, DSP, or GPU may execute a program stored in a memory to perform the encoding process of the present disclosure. Alternatively, the processor may be a dedicated circuit that performs signal processing on an audio signal, including the encoding process of the present disclosure.

[0169] The memory may be configured, for example, by RAM (Random Access Memory) or ROM (Read Only Memory). The memory may also include a magnetic storage medium such as a hard disk or a semiconductor memory such as an SSD (Solid State Drive). The term "memory" may also refer to an internal memory built into a CPU or GPU.

[0170] The communication IF (Interface) is a communication module compatible with a communication method such as Bluetooth (registered trademark) or WIGIG (registered trademark). The encoding device has a function of communicating with other communication devices via the communication IF and transmits an encoded bitstream.

[0171] The communication module is composed of, for example, a signal processing circuit and an antenna corresponding to the communication method. In the above example, Bluetooth (registered trademark) or WIGIG (registered trademark) is used as the communication method, but other communication methods such as LTE (Long Term Evolution), NR (New Radio), or Wi-Fi (registered trademark) may also be supported. Furthermore, the communication IF may be a wired communication method such as Ethernet (registered trademark), USB (Universal Serial Bus), or HDMI (registered trademark) (High-Definition Multimedia Interface) instead of the wireless communication method described above.

[0172] <Physical configuration of audio signal processing device> Fig. 14 is a diagram showing an example of the physical configuration of an audio signal processing device. Note that the audio signal processing device in Fig. 14 may be a decoding device. Furthermore, part of the configuration described here may be provided in the audio presentation device 602. Furthermore, the audio signal processing device shown in Fig. 14 is an example of the audio signal processing device 601 described above.

[0173] The acoustic signal processing device of FIG. 14 includes a processor, a memory, a communication IF, a sensor, and a speaker.

[0174] The processor may be, for example, a CPU (Central Processing Unit), a DSP (Digital Signal Processor), or a GPU (Graphics Processing Unit), and the CPU, DSP, or GPU may execute a program stored in a memory to perform the audio processing or decoding processing of the present disclosure. Alternatively, the processor may be a dedicated circuit that performs signal processing on audio signals, including the audio processing of the present disclosure.

[0175] The memory may be configured, for example, by RAM (Random Access Memory) or ROM (Read Only Memory). The memory may also include a magnetic storage medium such as a hard disk or a semiconductor memory such as an SSD (Solid State Drive). The term "memory" may also refer to an internal memory built into a CPU or GPU.

[0176] The communication IF (Interface) is a communication module compatible with a communication method such as Bluetooth (registered trademark) or WIGIG (registered trademark). The acoustic signal processing device shown in Fig. 14 has a function of communicating with other communication devices via the communication IF and acquires a bitstream to be decoded. The acquired bitstream is stored in a memory, for example.

[0177] The communication module is composed of, for example, a signal processing circuit and an antenna corresponding to the communication method. In the above example, Bluetooth (registered trademark) or WIGIG (registered trademark) is used as the communication method, but other communication methods such as LTE (Long Term Evolution), NR (New Radio), or Wi-Fi (registered trademark) may also be supported. Furthermore, the communication IF may be a wired communication method such as Ethernet (registered trademark), USB (Universal Serial Bus), or HDMI (registered trademark) (High-Definition Multimedia Interface) instead of the wireless communication method described above.

[0178] The sensor performs sensing to estimate the position or orientation of the listener. Specifically, the sensor estimates the position and / or orientation of the listener based on one or more detection results of the position, orientation, movement, velocity, angular velocity, or acceleration of a part or the entire body of the listener, such as the head, and generates position information indicating the position and / or orientation of the listener. Note that the position information may be information indicating the position and / or orientation of the listener in real space, or information indicating a displacement of the position and / or orientation of the listener based on the position and / or orientation of the listener at a predetermined time. Furthermore, the position information may be information indicating the position and / or orientation relative to the stereophonic sound reproduction system or an external device equipped with the sensor.

[0179] The sensor may be, for example, an imaging device such as a camera or a ranging device such as LiDAR (Light Detection and Ranging), and may capture an image of the listener's head movement and detect the movement of the listener's head by processing the captured image. Alternatively, the sensor may be a device that performs position estimation using wireless signals of any frequency band, such as millimeter waves.

[0180] The audio signal processing device shown in Fig. 14 may acquire position information from an external device equipped with a sensor via a communication IF. In this case, the audio signal processing device does not need to include a sensor. Here, the external device is, for example, the audio presentation device 602 described in Fig. 6 or a 3D video playback device worn on the listener's head. In this case, the sensor is configured by combining various sensors such as a gyro sensor and an acceleration sensor.

[0181] The sensor may, for example, detect the angular velocity of rotation around at least one of three mutually perpendicular axes in the sound space as the axis of rotation as the speed of movement of the listener's head, or may detect the acceleration of displacement with at least one of the three axes as the direction of displacement.

[0182] For example, the sensor may detect the amount of rotation about at least one of three mutually orthogonal axes in the sound space as the rotation axis, or the amount of displacement about at least one of the three axes as the displacement direction, as the amount of movement of the listener's head. Specifically, the sensor detects 6 DoF (position (x, y, z) and angle (yaw, pitch, roll)) as the position of the listener. The sensor is configured by combining various sensors used for detecting movement, such as a gyro sensor and an acceleration sensor.

[0183] The sensor may be any device capable of detecting the position of the listener, such as a camera or a GPS (Global Positioning System) receiver. Alternatively, the sensor may use location information obtained by performing self-position estimation using LiDAR (Laser Imaging Detection and Ranging). For example, when the audio signal reproduction system is implemented by a smartphone, the sensor is built into the smartphone.

[0184] The sensor may also include a temperature sensor such as a thermocouple that detects the temperature of the acoustic signal processing device shown in Figure 14, and a sensor that detects the remaining charge of a battery provided in or connected to the acoustic signal processing device.

[0185] A speaker has, for example, a diaphragm, a drive mechanism such as a magnet or a voice coil, and an amplifier, and presents an audio signal after acoustic processing to a listener as sound. The speaker operates the drive mechanism in response to the audio signal (more specifically, a waveform signal indicating the waveform of the sound) amplified by the amplifier, and the drive mechanism vibrates the diaphragm. In this way, the diaphragm vibrating in response to the audio signal generates sound waves, which propagate through the air and reach the listener's ears, causing the listener to perceive the sound.

[0186] Note that, although the description has been given here of an example in which the acoustic signal processing device shown in FIG. 14 includes a speaker and presents an audio signal after acoustic processing via the speaker, the means for presenting the audio signal is not limited to the above configuration. For example, the audio signal after acoustic processing may be output to an external audio presentation device 602 connected via a communication module. Communication via the communication module may be wired or wireless. As another example, the acoustic signal processing device shown in FIG. 14 may include a terminal for outputting an analog audio signal, and a cable such as earphones may be connected to the terminal to present the audio signal from the earphones. In the above case, the audio signal is reproduced by headphones, earphones, a head-mounted display, a neck speaker, a wearable speaker, a surround speaker composed of multiple fixed speakers, or the like, which is worn on the head or part of the body of the listener, which is the audio presentation device 602.

[0187] <Functional Description of Rendering Unit, Example 1> Examples 1 and 2 will be used to describe detailed configurations of the rendering units 1103 and 1202 in Fig. 11 and Fig. 12. Figs. 15 to 28 are diagrams for explaining specific examples of the sound reproduction system according to Example 1 of the embodiment.

[0188] In the first embodiment, the sound reaching the listener is compared with an evaluation value that takes into account the listening directionality, which is one of the listener's listening characteristics, and only a specified number of sounds are retained, while the remaining sound signals are reduced by at least one of culling processing and integration processing. Alternatively, the evaluation value is compared with a predetermined threshold, and sound signals with evaluation values ​​below the threshold are subjected to at least one of culling processing and integration processing. Below, an example of culling processing will be mainly described, and an example of integration processing will be described later.

[0189] Specifically, in this example, the evaluation value of each sound is corrected based on the listening directionality determined by the direction of the listener's face, by amplifying sounds coming from directly in front of the face and attenuating sounds coming from the dip direction. The listening directionality is designed in advance and stored as a table in a memory unit, or it is calculated using a binaural filter. By performing culling in this way while taking the listener's listening directionality into consideration, sounds that are less important to the listener (i.e., difficult to hear) are culled, rather than simply being based on the volume of the sound, thereby reducing the amount of processing while maintaining quality (sound quality).

[0190] 15 is a block diagram of a decoder according to this example, that is, a configuration of a rendering unit 1500. The basic idea of ​​this example is to cull sounds that reach the listener and are selected using an evaluation value that takes into account the listener's listening directionality, thereby reducing the number of filtering processes in the sound generation unit at the subsequent stage, thereby reducing the amount of processing (in other words, the amount of calculation).

[0191] First, input data (such as a bitstream) is provided to the spatial information management unit 1501. The input data includes an audio signal or encoded audio data representing an audio signal, and metadata used in acoustic processing. If encoded audio data is included, the encoded audio data is provided to an audio data decoder (not shown), which performs decoding processing to generate an audio signal. This audio signal is provided sequentially to the direct sound generation unit 1502, the reverberation sound generation unit 1503, the reflected sound generation unit 1504, and the diffracted sound generation unit 1505. If an audio signal is included instead of encoded audio data, the audio signal is provided sequentially to the direct sound generation unit 1502, the reverberation sound generation unit 1503, the reflected sound generation unit 1504, and the diffracted sound generation unit 1505. Note that providing sequentially means that the operation of providing the audio signal to one component and then providing it to the next component as an output from the component is performed continuously (i.e., sequentially). That is, in this configuration, the direct sound generation unit 1502, reverberation sound generation unit 1503, reflected sound generation unit 1504, and diffracted sound generation unit 1505 are connected in series, and the culling unit 1506 is arranged after each generation unit. With this configuration, the sound generated by the generation unit in the previous stage can affect the generation unit in the current stage, making it possible to provide accurate immersive audio that is closer to actual spatial acoustics.

[0192] In this figure, the configuration is such that all of the output of the diffracted sound generation unit 1505 is given to the culling unit 1506, but this is not limited to this, and the configuration may also be such that part of the output of the diffracted sound generation unit 1505 does not enter the culling unit 1506 but is output directly to the sound generation unit 1507.

[0193] The spatial information management unit 1501 extracts metadata from the input data, and the metadata is provided to a direct sound generation unit 1502 , a reverberation sound generation unit 1503 , a reflected sound generation unit 1504 , and a diffracted sound generation unit 1505 .

[0194] The structure of the metadata 1600 is shown in Fig. 16. Spatial information 1601 mainly represents information about the space in which immersive audio is provided to the listener, such as the shape of the room, characteristics of the wall materials (such as sound reflectance and absorption rate), characteristics of the obstacle materials (such as sound reflectance and absorption rate), and information about their placement. Object information 1602 mainly represents information about the position and orientation of sound source objects and the sounds emitted from the sound source objects. Listener information 1603 mainly represents information about the position and orientation of the listener.

[0195] The direct sound generation unit 1502, the reverberation sound generation unit 1503, the reflected sound generation unit 1504, and the diffracted sound generation unit 1505 each receive an audio signal and metadata, generate direct sound, reverberation sound, reflected sound, and diffracted sound, and output them to the culling unit 1507.

[0196] The culling unit 1506 identifies unimportant sounds from the signals input to the culling unit 1506, discards (or discards) the signals of the identified sounds, and outputs the remaining sounds (i.e., sounds that are important to the listener) to the sound generation unit 1507. Note that discarding a sound signal may also be expressed as bypassing or ignoring.

[0197] The sound generation unit 1507 performs acoustic filtering, such as convolving an HRTF (Head related transfer function), on the signal input from the culling unit 1506, and outputs the result as an output signal (output sound signal). This acoustic filtering is performed to provide the output signal to the listener in a form that is suitable for the output to the listener, such as headphones or multi-channel speakers.

[0198] FIG. 17 shows a conceptual diagram of the culling process in this example, that is, the operation of the culling unit 1506.

[0199] In the figure, the listener is facing diagonally upward to the left of the paper (the nose of the user 99 is positioned in front of the user 99), and the listening directionality of this listener is such that the sensitivity is high in the front direction of the face and low in the back direction of the head, as shown by the dashed dotted line. Also, it is assumed that the direct sound (a) from the sound source object 98 reaches the listener, and also the reflected sound (b), the reverberated sounds (c) to (g), and the diffracted sound (h) that has passed through an obstacle 97 reach the listener.

[0200] If listening directionality is not taken into consideration, the sounds to be culled are determined simply by comparing the evaluation values ​​of the sounds that reach the listener. On the other hand, if listening directionality is taken into consideration, as shown by the dashed-dotted line in the figure, the listening directionality is strong (high) in the direction the face is facing, and weak (low) in the direction of the back of the head. Therefore, sounds that reach the front of the listener, such as (a) direct sound, (b) reflected sound, (c) reverberation sound, and (h) diffracted sound, tend to remain without being culled, while (d) to (g) reverberation sounds that reach the sides or back of the listener's head are more likely to be selected as sounds to be culled. Since listening directionality indicates the ease of hearing from the direction in which sounds reach the listener, it is realistic that sounds that arrive from the direction in which the listener is facing tend to remain.

[0201] The evaluation value of a sound reaching a listener when listening directionality is taken into consideration can be expressed, for example, by multiplying the strength of the sound reaching the listener or the strength of the sound after auditory correction by a weight corresponding to the listening directionality. Using the example in this figure, the greatest weight is given to the reflected sound (b) arriving from the direction with the strongest listening directionality, and conversely, the least weight is given to the reverberant sound (f) arriving from the direction with the weakest listening directionality. In this way, an evaluation value taking listening directionality into consideration is calculated, and the evaluation values ​​of each sound reaching the listener are compared to determine the sounds to be culled.

[0202] Here, the listening directionality may be designed based on the shape of the listener above the neck and stored in the decoder as a predetermined table, or may be determined by analyzing the HRTF filter (or binaural filter) used in the sound generation unit 1507.

[0203] The operation of the decoder shown in the figure is carried out as shown in FIG.

[0204] First, a determination unit (not shown) determines whether metadata is input (i.e., whether metadata is input) (S1801). If metadata is input (Yes in S1801), the process proceeds to generation of direct sound, etc., and if metadata is not input (No in S1801), the process ends.

[0205] Once the metadata is input (Yes in S1801), the sounds that reach the listener, i.e., direct sound, are generated (S1802), reverberant sound is generated (S1803), reflected sound is generated (S1804), and diffracted sound is generated (S1805).

[0206] Next, the evaluation value (sound intensity or sound intensity corrected for auditory perception) of each sound reaching the listener is calculated (S1806), and then the evaluation value of each sound reaching the listener is multiplied by a weight corresponding to the listening directionality (S1807).

[0207] The sounds to be culled are determined based on the evaluation value of each sound after multiplication by a weight corresponding to the listening directionality (S1808). For example, the evaluation values ​​after weight multiplication (weighted evaluation values) are compared, and only a specified number of sounds with the highest weighted evaluation values ​​are retained, while the other sounds are culled. Alternatively, the weighted evaluation value is compared with a predetermined threshold, and sounds below the threshold are culled. The remaining sounds that have not been culled are output to the sound generation unit 1507.

[0208] As a result of the culling process, the remaining direct sound, reverberant sound, reflected sound, and diffracted sound are subjected to stereophonic signal processing such as HRTF to generate a stereophonic signal (S1809), which is output to a driver in the device used by the listener, such as headphones.

[0209] Then, the process returns to step S1801, where it is determined whether new metadata is to be input.

[0210] Although an example of a decoder configured in series has been described above, a decoder may also be configured in parallel, for example, as shown in Fig. 19. In the decoder configuration of Fig. 19, a direct sound generation unit 1502, a reverberation sound generation unit 1503, a reflected sound generation unit 1504, and a diffracted sound generation unit 1505 are configured in parallel, and the sounds generated by each generation unit are evaluated and culled. Here, the configuration is such that the output signals of all generation units are provided to a culling unit 1506, but this is not limiting, and the output of some generation units may be output directly to a sound generation unit 1507 without entering the culling unit 1506.

[0211] Although the above describes a decoder that performs culling processing, it is also possible to perform integration processing, for example, as shown in FIGS. 20A and 20B . FIG. 20A is a block diagram of a decoder configuration when integration processing is performed. In this configuration, the rendering unit 2000 includes an integration unit 2001, instead of the culling unit 1507 that performs culling processing, that integrates sounds and reduces the number of filtering processes performed by the downstream sound generation unit 1507, thereby reducing the amount of calculations while maintaining the quality of immersive audio. Note that components having the same functions as those in FIG. 15 are assigned the same reference numerals and will not be described here. The operation of the integration unit 2001 will be further described in Example 2. The decoder shown in FIG. 20A operates as shown in FIG. 20B . As shown in the figure, the flow in this example differs from the flow shown in FIG. 18 in that steps S2001 and S2002 are performed instead of steps S1806 and S1808. That is, in this example, after the decoder generates the sounds that will reach the listener, it calculates the intersection angle of any two sounds that will reach the listener (S2001). Then, it performs integration processing based on a value corresponding to the magnitude of this intersection angle. As a specific example, it compares each intersection angle with a threshold value for angle discrimination ability, integrates two sounds that have an intersection angle smaller than the threshold value (i.e., located at a narrow angle), constructs a virtual object that emits a virtual sound, and generates the sound of the virtual object (S2002).

[0212] As a result of this integration process, the remaining direct sound, reverberant sound, reflected sound, diffracted sound, and virtual sound of the virtual object are subjected to stereophonic signal processing such as HRTF to generate a stereophonic signal (S1809), which is then output to a driver in the device used by the listener, such as headphones.

[0213] It should be noted that the decoder (rendering unit 2100) having the integration unit 2001 can also be configured in parallel as shown in FIG.

[0214] Here, Figures 22 to 24 are figures for explaining the benefits of outputting sounds that reach listeners selected using evaluation values ​​based on listening directionality by integrating them rather than culling them.

[0215] When using the integration unit 2001, instead of culling sounds reaching a listener selected using an evaluation value based on listening directionality, the culled sounds are output as virtual sounds represented by fewer virtual objects than the sounds to be culled. This is because listening directionality is sensitive to the direction of the listener's face, and when the listener (user 99) moves their face, as shown in the changes from FIG. 22 to FIG. 23, the sounds selected for culling (sounds indicated by crossed arrows in the figure) tend to fluctuate, causing the direction of the sounds reaching the listener to frequently change, resulting in unnatural immersive audio for the listener. To avoid this problem, instead of performing culling as shown in FIG. 24, sounds are represented by a small number of virtual objects 96. This avoids the problem of the direction of the sounds reaching the listener frequently changing when virtual sounds from the virtual objects 96 are output, and reduces degradation in the quality of the immersive audio.

[0216] In these figures, for the face orientation shown in FIG. 22, the reflected sounds (e) to (g) are selected for culling, while for the face orientation shown in FIG. 23, the reflected sound (b) and the reflected sounds (c) to (e) are selected for culling.

[0217] As shown in these figures, when the direction of the face changes, the reverberation sounds (e) to (g) disappear at one point, and the reflected sound (b) and the reverberation sounds (c) to (e) disappear at another point due to culling, causing the direction of the sound heard by the listener to frequently switch, causing the listener to perceive a deterioration in the quality of the immersive audio.

[0218] Therefore, by integrating multiple sounds that would otherwise be culled and presenting them to the listener as virtual object 96 (in Figure 24, sounds (b + c), (d + e), and (f + g) are each integrated into virtual object 96), no switching of sounds occurs due to culling, and the listener does not perceive any degradation in the quality of the immersive audio.

[0219] One method for combining multiple sounds to be culled is to add together the signals of the sounds to be culled and regard the resulting sound as a sound output from the virtual object 96. When adding the sounds together, at least one of the energy and phase of each sound may be adjusted before the addition.

[0220] 25 to 28 are characterized in that the listening directionality used in this example is expressed in 360-degree directions, up and down (i.e., vertical direction) and left and right (i.e., horizontal direction). This model representing 360-degree directions, up and down and left and right, will hereinafter be referred to as a 3D spherical model.

[0221] Figure 25 shows a conceptual diagram of listening directionality represented by a 3D spherical model. As shown in this figure, the listening characteristics of a typical listener are often such that sensitivity is high in the front and low in the rear, above, and below. Figures 27 and 28 show examples of listening directionality projected onto the X-Y plane, the X-Z plane, and the Y-Z plane in Figure 25, respectively.

[0222] When this type of listening directionality is used, sounds that reach the listener from the front are less likely to be culled, while sounds that reach the listener from behind, above, and below are more likely to be culled. Also, sounds that reach the sides are more likely to be culled somewhere between the front and the back.

[0223] The above description has been given based on Example 1, but Example 1 is not limited to the above description. For example, it is also possible to use listening directivity that takes into account the influence of the listener's face and head shape, hairstyle, and accessories (i.e., listening directivity with a shape different from that shown in FIG. 25 ). As an example, the actual target listener's face and head shape, hairstyle, and accessories such as hats may be converted into data, and the listening directivity may be designed using this data, taking into account the influence of these shapes and materials. This makes it possible to use listening directivity that is more suited to the actual situation, suppressing degradation of immersive audio quality due to culling, and reducing the amount of calculations.

[0224] The present invention can also be applied to automatic gain control (AGC). For example, while the above example describes the application of the listener's listening directionality to culling, the essence of the present invention, i.e., an input signal correction technology that utilizes the listener's listening directionality, can be applied to other technical fields. An example of such an application is automatic gain control (AGC). AGC is a technology that automatically multiplies an input signal by a gain to stabilize the signal level to a predetermined level when the input signal level (energy) is low, making the input signal easier to listen to. When calculating the gain, the gain is calculated based on the listener's listening directionality. If the input signal arrives from a direction with high listening directionality sensitivity, the gain multiplied by the input signal is reduced, and if the input signal arrives from a direction with low listening directionality sensitivity, the gain multiplied by the input signal is increased. This has the advantage of stabilizing the listening level of the sound reaching the listener, making listening easier. One possible application of this technology is hearing aids. Hearing aids receive input signals using a combination of multiple directional microphones. The combination of these directional microphones forms a listening directionality, and by regarding this listening directionality as the listening directionality of the listener, it becomes possible to apply the essence of the present invention.

[0225] In addition to sound, the present invention can also be applied to light, computer vision, and other fields. While the present invention has been described based on sound propagation in this example, at least some of the techniques of this example can be applied to light propagation, for example. Regarding light propagation, the present invention is applicable to computer graphics that generate scenes based on direct light, reflected light, and diffracted light. Specifically, the present invention culls light reaching the user using light (direct light, reflected light, and diffracted light) reaching the user from a light source in a virtual space or a space that combines virtual space and real space. When performing culling, the user's visual characteristics are taken into consideration, and an evaluation value of the light reaching the user is calculated using a weighting corresponding to the visual characteristics. Light reaching the user that is subject to culling is selected by comparing the evaluation values ​​with each other or with a threshold. This minimizes the degradation in the quality of computer graphics presented to the user due to culling, and significantly reduces the amount of computation required to generate computer graphics.

[0226] <Functional Description of Rendering Unit, Example 2> Figures 29 to 56 are diagrams for explaining a specific example of an audio reproduction system according to Example 2 of an embodiment. The device configuration of the rendering unit in this example is the same as that shown in any one of Figures 15, 19, 20A, and 21, and therefore will not be described here.

[0227] In this example, for sounds that reach the listener, at least one of sounds to be culled and sounds to be integrated is determined based on the relationship between two or more sounds and the listener's hearing characteristics, and culling or sound integration is performed on the determined sounds. For example, if the hearing characteristics are the listener's angle discrimination ability, culling or integration processing is performed based on a value corresponding to the magnitude of the angle between the sounds that reach the listener. As one specific example, sounds are culled or integrated based on whether the angle between the sounds that reach the listener falls within a threshold of the listener's angle discrimination ability (the angle at which sounds can be distinguished as different sounds).

[0228] A conceptual diagram of the sound reduction process in this example is shown in Fig. 29. This diagram shows a listener (user 99), a sound source object 98, and an obstacle 97 arranged in a room, as viewed from above the room.

[0229] In the figure, the listener is facing diagonally upward to the left of the page, and the listener's hearing characteristics (in this embodiment, angular discrimination ability) are shown by dashed lines extending radially from the listener. The angle between two adjacent dashed lines in the angular discrimination ability represents the threshold at which the listener can distinguish between two sounds reaching the listener. If the intersection angle (angle between the two sounds) is smaller than this threshold (narrow angle), the listener will be unable to distinguish between the two sounds and will perceive that a single sound is reaching them.

[0230] In this diagram, for convenience, the listener's angular discrimination ability is shown in a fixed direction, but in reality it does not have to be a fixed direction. Whether the listener can distinguish the difference between the two sounds is determined by comparing the intersection angle of the two sounds reaching the listener with the threshold value of the angular discrimination ability (the angle between two adjacent dotted lines).

[0231] In the figure, the angle of intersection between the direct sound (a) and the diffracted sound (h) is smaller than the threshold, and the angle of intersection between the reverberant sound (d) and the reverberant sound (e) is also smaller than the threshold. In such a case, the listener cannot distinguish between at least one of the direct sound (a) and the diffracted sound (h), so at least one of the direct sound (a) and the diffracted sound (h) is culled (in the figure, the diffracted sound (h) is culled). Similarly, the listener cannot distinguish between at least one of the reverberant sound (d) and the reverberant sound (e), so at least one of the reverberant sound (d) and the reverberant sound (e) is culled (in the figure, the reverberant sound (d) is culled).

[0232] The sound to be culled here may simply be the lower level of the two sounds, or the sound to be culled may be determined taking into account human hearing characteristics (for example, by using the level after weighting according to hearing directionality).

[0233] The operation of the decoder shown in the figure is carried out as shown in FIG.

[0234] First, a determination unit (not shown) determines whether metadata is input (i.e., whether metadata has been input) (S3001). If metadata has been input (Yes in S3001), the process proceeds to the generation of direct sound, etc., and if metadata has not been input (No in S3001), the process ends.

[0235] Once the metadata is input (Yes in S3001), the sounds that reach the listener, i.e., direct sounds, are generated (S3002), reverberant sounds are generated (S3003), reflected sounds are generated (S3004), and diffracted sounds are generated (S3005).

[0236] Next, the intersection angle between any two sounds reaching the listener is calculated (S3006). After that, each intersection angle is compared with a threshold value of angle discrimination ability, and at least one of the two sounds with an intersection angle smaller than the threshold value is culled (S3007).

[0237] As a result of the culling process, the remaining direct sound, reverberant sound, reflected sound, and diffracted sound are subjected to stereophonic signal processing such as HRTF to generate a stereophonic signal (S3008), which is output to a driver in the device used by the listener, such as headphones.

[0238] Then, the process returns to step S3001, where it is determined whether new metadata is to be input.

[0239] The effect of reducing the amount of calculation according to the present invention will be explained below. According to the present invention, (1) the amount of calculation increases by the amount of processing for identifying sounds to be culled, and (2) the amount of calculation for processing such as HRTF convolution, which becomes unnecessary, is reduced by the number of sounds to be culled.

[0240] Below, the amounts of calculation for (1) and (2) are compared using the following specific set values.

[0241] The conditions for comparing the amount of calculations are: signal sampling rate: 48 kHz, HRTF length: 10 ms, number of sounds reaching the listener: 50, and update period for parameters such as angle: 200 ms. The amount of calculations is assumed to be one operation each for addition / subtraction, multiplication, and product-sum operations, and 25 operations for function operations. For convenience, the calculation of the angle between two vectors, comparison, and other processing are assumed to require 100 operations.

[0242] (1) Process for identifying sounds to be culled When 50 sounds reach the listener, any two-tone combination of those 50 sounds can be selected, resulting in 225 combinations. Since the parameter update period is 200 ms, there are five updates per second. The amount of computation required for the process to identify sounds to be culled is calculated in MOPS (Million Operations per Second) as 1225 x 5 x 100 / 1000000 = approximately 0.6 MOPS.

[0243] (2) Processing such as convolution of HRTF for the number of sounds targeted for culling: Assume that sounds reaching one listener are culled according to the present invention. Since the length of the HRTF is 10 ms, the filter order of the HRTF is 480. To convolve the HRTF into a signal with a sampling rate of 48 kHz, the amount of calculation is 48,000 x 480 x 1 / 1,000,000 = approximately 23 MOPS.

[0244] Therefore, under the above conditions, application of the present invention has the effect of reducing the amount of calculation by approximately 22.4 MOPS. Note that the explanation given here is merely an example, and it is clear that the effect of reducing the amount of calculation will naturally change if the conditions change.

[0245] The concept of sound integration processing in this example will now be described with reference to Figures 31 and 32. Figures 31 and 32 show diagrams similar to Figure 29.

[0246] In Figure 31, the angle of intersection between the direct sound (a) and the diffracted sound (h) is smaller than a threshold, and the angle of intersection between the reverberant sound (d) and the reverberant sound (e) is also smaller than a threshold. In such a case, the listener cannot distinguish between at least one of the direct sound (a) and the diffracted sound (h), so the direct sound (a) and the diffracted sound (h) are integrated. Similarly, the listener cannot distinguish between at least one of the reverberant sound (d) and the reverberant sound (e), so the reverberant sound (d) and the reverberant sound (e) are integrated. Figure 32 shows how multiple sounds are integrated to form a virtual object and how sound is output from the virtual object.

[0247] By presenting multiple sounds to be integrated together as an output signal from a virtual object to the listener (in Figure 32, the direct sound (a), the diffracted sound (h), the reverberant sound (d), and the reverberant sound (e) are each integrated into a virtual object), the listener is less likely to perceive a degradation in the quality of the immersive audio than when culling is performed.

[0248] Note that, as a method for integrating multiple sounds to be integrated, for example, a method of adding the sounds to be integrated and treating them as a sound output from a virtual object can be mentioned. When adding the sounds, at least one of the energy and phase of each sound may be adjusted before adding them. Note that the method given here is merely an example, and the method for integrating multiple sounds is not limited to this method.

[0249] 33, the position of the virtual object is either an area (hatched area) surrounded by the direction connecting the listener and one sound (dotted circle) or the direction connecting the listener and the other sound (dotted circle). Alternatively, the position of the virtual object may be either an area surrounded by a direction obtained by softening outward the direction connecting the listener and one sound or a direction obtained by softening outward the direction connecting the listener and the other sound, as shown in FIG.

[0250] Furthermore, sound culling may be performed on the sounds that reach the listener based on the level ratio between the sounds that reach the listener.

[0251] The advantage of this example is that, among the sounds that reach the listener, sounds are culled based on the level ratio between the sounds that reach the listener, and the number of filtering processes in the subsequent sound generation unit 1507 is reduced, thereby reducing the amount of calculation while maintaining the quality of immersive audio.

[0252] This conceptual diagram is shown in Figure 35. The sounds that reach the listener include the direct sound (a), reflected sound (b), reverberant sounds (c) to (g), and diffracted sound (h) through an obstacle 97, with the volume (level) of each sound being as shown in the figure. The threshold for the level ratio at which culling is performed is -26 dB. That is, if the louder of two sounds has a level of 70 dB and the quieter sound has a level of 50 dB, the level ratio of the two (calculated as the difference between the quieter sound and the louder sound in the logarithmic (dB) domain) is 50 - 70 = -20 dB, which exceeds the threshold of -26 dB. In this case, the quieter sound is not culled.

[0253] On the other hand, if the level of the larger of the two sounds is 70 dB and the level of the smaller sound is 30 dB, the level ratio between the two sounds is 30-70=-40 dB, which is below the threshold of -26 dB. In this case, the smaller sound is culled.

[0254] In the situation shown in the figure, the level of the reflected sounds (d) to (g) is below the threshold of -26 dB relative to the level of the direct sound (a). Also, the level of the reflected sound (f) is below the threshold of -26 dB relative to the reflected sound (b). Therefore, the reflected sounds (d) to (g) are masked by the direct sound (a), and the reflected sound (f) is masked by the reflected sound (b), so the listener cannot perceive these sounds, and so these sounds are culled.

[0255] Furthermore, this sound level is based on the assumption that signal energy across all bands is used, but this is not limiting; the sound to be culled may also be determined using signal energy that utilizes human hearing characteristics (for example, by calculating energy by assigning a heavy weight to bands that are important to the auditory sense).

[0256] The sound level may also be calculated based on the level ratio (difference in the logarithmic domain) for each subband of the two sounds. This is because human hearing characteristics vary with respect to the frequency axis, and the level ratio (difference in the logarithmic domain) for each subband of the two sounds can be regarded as a method of calculating signal energy that takes into account human hearing characteristics.

[0257] The figure shows an example in which the threshold of the masking effect is constant at -26 dB regardless of the intersection angle of the two sounds that reach the listener. It is known that the threshold of the masking effect changes depending on the intersection angle of the two sounds that reach the listener. Specifically, when the intersection angle of the two sounds that reach the listener is small, the masking effect is large, and when the intersection angle of the two sounds is large, the masking effect is small.

[0258] The feature of this example is that among the sounds that reach the listener, sounds are culled based on a threshold determined by the intersection angle between the sounds that reach the listener and the level ratio. The threshold is determined so that culling becomes less likely when the intersection angle becomes larger, or when the level ratio between the two signals becomes larger. This is a model of human hearing characteristics. Figure 36 shows a diagram similar to Figure 35. Note that the angle of incidence of each sound is expressed as 360 degrees counterclockwise, with the direction in which the face is facing forward being 0 degrees.

[0259] The level ratio at which culling is performed is determined by the intersection angle of the two sounds as follows:

[0260] Intersection angle is 0 degrees or more and less than 45 degrees: threshold = -22 dB Intersection angle is 45 degrees or more and less than 90 degrees: threshold = -26 dB Intersection angle is 90 degrees or more and less than 135 degrees: threshold = -30 dB Intersection angle is 135 degrees or more and less than 180 degrees: threshold = -34 dB

[0261] In this way, the larger the intersection angle between the two sounds, the lower the threshold value, and culling becomes difficult if the level ratio between the two signals is not large.

[0262] For example, when considering the direct sound (a) and the reflected sound (b), the intersection angle between them is 40 degrees, and the threshold at this time is -22 dB. The level ratio between the direct sound (a) and the reflected sound (b) is -10 dB, which exceeds the threshold, so culling is not performed. On the other hand, when considering the direct sound (a) and the diffracted sound (h), the intersection angle between them is 15 degrees, and the threshold at this time is -22 dB. The level ratio between the direct sound (a) and the diffracted sound (h) is -25 dB, which is below the threshold, so culling is performed. In this case, the diffracted sound (h), which has a low level, is culled.

[0263] In this way, all sound combinations are judged, and the sounds to be culled are identified. In the example shown in the figure, the sounds to be culled are the reverberant sounds (e) to (g) and the diffracted sound (h).

[0264] By integrating sounds based on both the intersection angle and level ratio of the sounds that reach the listener in this way, it becomes possible to determine the standard for the level ratio according to the intersection angle, or vice versa, to determine the standard for the intersection angle according to the level ratio, making it possible to more effectively reduce the amount of calculation while maintaining the quality of immersive audio.

[0265] Here, it is assumed that the sound level is determined using signal energy across all bands, but this is not limiting; the sound to be culled may also be determined using signal energy that utilizes human hearing characteristics (for example, by calculating energy by assigning a heavy weight to bands that are important to the auditory sense).

[0266] The sound level may also be calculated based on the level ratio (difference in the logarithmic domain) for each subband of the two sounds. This is because human hearing characteristics vary with respect to the frequency axis, and the level ratio (difference in the logarithmic domain) for each subband of the two sounds can be regarded as a method of calculating signal energy that takes into account human hearing characteristics.

[0267] Next, we will explain a configuration that reduces the amount of calculation while maintaining the quality of immersive audio by integrating sounds that reach the listener based on the level ratio between the sounds that reach the listener and reducing the number of filtering processes in the sound generation unit at the subsequent stage. This conceptual diagram is shown in Figure 37. Figure 37 shows a configuration similar to that of Figure 35.

[0268] Here, the level ratio for integration is -26 dB. In other words, if the level of the larger of two sounds is 70 dB and the level of the smaller sound is 50 dB, the level ratio between the two (calculated in the logarithmic (dB) domain as the difference between the smaller and larger sounds) is 50 - 70 = -20 dB, which exceeds the threshold of -26 dB. In this case, the smaller sound is not integrated.

[0269] On the other hand, if the level of the larger of the two sounds is 70 dB and the level of the smaller sound is 30 dB, the level ratio of the two sounds is 30-70=-40 dB, which is below the threshold of -26 dB. In this case, the smaller sound is integrated.

[0270] In the situation shown in the figure, the level of the reflected sounds (d) to (g) is below the threshold of -26 dB relative to the level of the direct sound (a). Furthermore, the level of the reflected sound (f) is below the threshold of -26 dB relative to the reflected sound (b). Therefore, due to the masking effect, the listener cannot perceive the reflected sounds (d) to (g) caused by the direct sound (a), or the reflected sound (f) caused by the reflected sound (b). Therefore, the reflected sounds (d) to (g) are integrated to generate a virtual object. This reduces the number of sounds processed by the sound generation unit 1507 at the subsequent stage, thereby reducing the amount of calculations.

[0271] Furthermore, this sound level is based on the assumption that signal energy across all bands is used, but this is not limiting; the integrated sound may also be determined using signal energy that utilizes human hearing characteristics (for example, by calculating energy by assigning a large weight to bands that are important to the auditory sense).

[0272] The sound level may also be calculated based on the level ratio (difference in the logarithmic domain) for each subband of the two sounds. This is because human hearing characteristics vary with respect to the frequency axis, and the level ratio (difference in the logarithmic domain) for each subband of the two sounds can be regarded as a method of calculating signal energy that takes into account human hearing characteristics.

[0273] In the example shown in the figure, the sounds to be integrated (indicated by the circular arrows in the figure, and the same applies to subsequent figures) are four, including the reverberant sounds (d) to (g), so the number of virtual objects will be fewer than the sounds to be integrated, i.e., between one and three.

[0274] There are various methods for constructing a virtual object, for example, a virtual object may be generated using only the sounds to be integrated, or a virtual object may be generated including the sounds to be integrated and sounds in their vicinity. Representative examples of the various methods for constructing a virtual object are shown in Figures 38 to 43.

[0275] Furthermore, as a method for integrating multiple sounds to be integrated, for example, the sounds to be integrated are added together and considered as a sound output from a virtual object. When adding the sounds together, at least one of the energy and phase of each sound may be adjusted before adding them. The method given here is merely an example, and the method for integrating multiple sounds is not limited to this method.

[0276] Next, Fig. 44 will be described. This example is characterized in that, among the sounds that reach the listener, sounds are integrated based on a threshold determined by the intersection angle between the sounds that reach the listener and a level ratio. The threshold is determined so that the larger the intersection angle, the more difficult it is to integrate unless the level ratio of the two signals is large. This is a model of human hearing characteristics. Fig. 44 shows a diagram similar to Fig. 35. Note that the angle of incidence of sound is expressed as 360 degrees counterclockwise, with the direction in which the face is facing forward being 0 degrees.

[0277] The level ratio at which culling is performed is determined by the intersection angle of the two sounds as follows:

[0278] Intersection angle is 0 degrees or more and less than 45 degrees: threshold = -22 dB Intersection angle is 45 degrees or more and less than 90 degrees: threshold = -26 dB Intersection angle is 90 degrees or more and less than 135 degrees: threshold = -30 dB Intersection angle is 135 degrees or more and less than 180 degrees: threshold = -34 dB

[0279] In this way, the larger the intersection angle between the two sounds, the lower the threshold value becomes, and unless the level ratio between the two signals is large, it becomes difficult to perform integration.

[0280] For example, when considering the direct sound (a) and the reflected sound (b), the intersection angle between them is 40 degrees, and the threshold at this point is -22 dB. The level ratio between the direct sound (a) and the reflected sound (b) is -10 dB, which exceeds the threshold and therefore integration is not performed. On the other hand, when considering the direct sound (a) and the diffracted sound (h), the intersection angle between them is 15 degrees, and the threshold at this point is -22 dB. The level ratio between the direct sound (a) and the diffracted sound (h) is -25 dB, which is below the threshold and therefore integration is performed.

[0281] In this way, all sound combinations are judged, and the sound to be integrated is identified. In the example shown in the figure, the sounds to be integrated (indicated by the circular arrows) are the reverberant sounds (e) to (g) and the diffracted sound (h).

[0282] In this way, by integrating sounds based on both the intersection angle and level ratio between the sounds that reach the listener, it becomes possible to determine the standard for the level ratio according to the intersection angle, or conversely, to determine the standard for the intersection angle according to the level ratio, making it possible to more effectively reduce the amount of calculation while maintaining the quality of immersive audio.

[0283] Here, it is assumed that the sound level is determined using signal energy across all bands, but this is not limiting; the integrated sound may also be determined using signal energy that utilizes human hearing characteristics (for example, by calculating energy by assigning a large weight to bands that are important to the auditory sense).

[0284] The sound level may also be calculated based on the level ratio (difference in the logarithmic domain) for each subband of the two sounds. This is because human hearing characteristics vary with respect to the frequency axis, and the level ratio (difference in the logarithmic domain) for each subband of the two sounds can be regarded as a method of calculating signal energy that takes into account human hearing characteristics.

[0285] Next, Figures 45 to 49 will be described. These figures show that the listener's ability to discriminate angles when viewed horizontally shows that the angle resolution is fine in the direction straight ahead of the face, but becomes coarser as one moves from the side to the rear.

[0286] Figures 45 and 46 show the relationship between the listener's orientation and 3D coordinates. Note that when the X-Y plane is viewed from above in Figure 45, a horizontal view is seen as shown in Figure 46. As shown in Figure 46, angle discrimination is used, with finer angle resolution in the frontal direction of the face and coarser angle resolution as one moves from the side to the rear. This makes it difficult for culling or integration of sounds in directions with high sensitivity to be performed, and makes it easier for culling or integration of sounds in directions with low sensitivity to be performed. Therefore, culling or integration of sounds in directions with low sensitivity is performed, reducing the amount of calculation while maintaining the quality of immersive audio.

[0287] 47 to 49, the listener's angular discrimination ability in the horizontal direction (X-Y plane) is high (high resolution), but the listener's angular discrimination ability in the vertical direction (Y-Z plane and X-Z plane) is low (low resolution). This makes it difficult to cull or integrate sounds in directions with high sensitivity, and makes it easier to cull or integrate sounds in directions with low sensitivity. Therefore, culling or integration of sounds in directions with low sensitivity is performed, making it possible to reduce the amount of calculation while maintaining the quality of immersive audio.

[0288] Next, we will explain Figures 50 and 51. In Figure 50, the reverberation sound (d) and the reverberation sound (e) are selected as sounds to be integrated (circular arrows). In this case, as already described, the reverberation sounds (d) to (e) are integrated to form a virtual object, and sound is output from the virtual object toward the listener. On the other hand, the example in Figure 51 is different in that in addition to the reverberation sounds (d) to (e), nearby reverberation sounds (c) and (f) are also used to form a virtual object, and sound is output toward the listener.

[0289] The reason why a virtual object is configured to include not only the sounds selected as targets for integration but also nearby sounds that are not selected as targets is to avoid changes in the sound output from the virtual object, which would occur if the sounds selected as targets for integration change over time when the object or the listener moves (for example, changing from reverberation sounds (d) to (e) to reverberation sounds (e) to (f)). When such a change in the virtual object occurs, the position where the sound is generated or the characteristics of the generated sound may suddenly change, and in that case the listener may perceive a deterioration in the quality of the immersive audio.

[0290] On the other hand, as in the example shown in this figure, by configuring a virtual object that includes not only the sounds selected as targets for integration but also nearby sounds, even if the object or listener moves, the range of sounds to be integrated is wide, so the virtual object will be configured to include sounds that will likely be selected at the destination. Therefore, changes to the virtual object are less likely to occur, which has the effect of reducing the frequency of the degradation in immersive audio quality described above.

[0291] Next, we will explain Figures 52 to 56. Figures 52 and 53 show diagrams similar to Figure 35. In Figure 52, before the listener moves, reverberation sounds (d) to (e) are selected as sounds to be integrated (circular arrows). Then, after a certain amount of time has passed and the listener moves, reverberation sounds (e) to (f) are selected, as indicated by the circular arrows in Figure 53.

[0292] At this time, for Fig. 52, a virtual object is constructed based on the reverberation sounds (d) to (e), and the sound generated by the virtual object is output to the listener, as shown in Fig. 54. On the other hand, for Fig. 53, a virtual object is constructed based on the reverberation sounds (e) to (f), and the sound generated by the virtual object is output to the listener, as shown in Fig. 55.

[0293] At this time, a change occurs in the position and characteristics of the sound from the reverberations (d) to (e) generated by the virtual object to the reverberations (e) to (f), and the listener may feel as if the position and characteristics of the sound have suddenly changed, resulting in a perceived degradation in the quality of the immersive audio.

[0294] To solve this problem, in this example, processing is added so that the position and characteristics of the sound gradually change, thereby mitigating the degradation of immersive audio quality.

[0295] Specifically, as shown in Fig. 56, the end portions of the reverberations (d) to (e) generated by the virtual object before the listener moves and the beginning portions of the reverberations (e) to (f) generated by the virtual object after the listener moves are generated so as to overlap in time, and these are multiplied by the corresponding window functions and added together to generate the sound that is finally output to the listener. Here, the window functions for the reverberations (d) to (e) have a shape that gradually attenuates, and the window functions for the reverberations (e) to (f) have a shape that gradually amplifies.

[0296] Furthermore, the position of the virtual object is controlled so that it changes gradually from the position of the virtual object before the listener moves to the position of the virtual object after the listener moves, as shown in FIGS. 54 and 55 .

[0297] By implementing this process, changes in the position and characteristics of the sound are smoothed, preventing degradation of the quality of the immersive audio, ultimately providing high-quality immersive audio to the listener.

[0298] The above description has been given based on Example 2, but Example 2 is not limited to the above description. For example, sounds to be culled or integrated may be determined using the intersection angle and level ratio of two sounds that reach the listener. In the above embodiment, a technology for changing the threshold for the intersection angle of two sounds according to the level ratio of two sounds that reach the listener has been described, but the present invention is not limited to this. Sounds to be culled or integrated may be determined using another determination method that combines the intersection angle and level ratio of two sounds. For example, there is a method for changing the threshold for the level ratio of two sounds according to the intersection angle of two sounds.

[0299] Furthermore, for example, sounds to be subject to culling or integration may be determined based on the distance between the listener and two sounds reaching the listener. A threshold related to hearing characteristics is used that makes sounds more likely to be subject to culling or integration the farther away from the listener the positions of the two sounds reaching the listener (the position of an object for direct sounds, or the position of the wall or obstacle last encountered for reflected, reverberant, or diffracted sounds). (If the hearing characteristic is angular discrimination ability, the angle threshold is widened, and if the hearing characteristic is auditory masking, the level ratio threshold is increased.) This makes sounds that reach the listener from far away from the listener more likely to be subject to culling or integration, thereby reducing the amount of calculations while minimizing degradation in the sound quality of immersive audio.

[0300] Furthermore, when two sounds reach a listener, one of which is farther away from the listener than the other, the sound that is farther away from the listener can be made easier to cull, thereby reducing the number of sounds that reach the listener, or the two sounds can be made easier to integrate, thereby reducing the number of sounds that reach the listener. This takes advantage of the fact that the sound that is farther away from the listener is harder for the listener to hear than the other sound, that is, it has less impact on the listener. By performing such culling or integration, it is possible to reduce the amount of calculation while minimizing degradation in the sound quality of immersive audio.

[0301] Alternatively, sounds to be culled or integrated may be determined based on the levels of two sounds reaching the listener. A threshold related to the hearing characteristic is used, which makes sounds more likely to be targeted for culling or integration the lower the levels of the two sounds reaching the listener (if the hearing characteristic is angular discrimination ability, the angle threshold is widened, and if the hearing characteristic is auditory masking, the level ratio threshold is increased). This makes sounds that reach listeners with low levels more likely to be targeted for culling or integration, thereby reducing the amount of calculations while minimizing degradation in the sound quality of immersive audio.

[0302] Furthermore, sounds to be subject to culling or integration may be determined depending on the output device used by the listener for listening. The threshold for culling or integration is changed depending on whether the output device used by the listener for listening is headphones or speakers. For example, the threshold for hearing characteristics may be changed so that headphones are less likely to be subject to culling or integration, or vice versa. Depending on the listening environment of the listener, the sensitivity to sound quality degradation of immersive audio changes depending on whether the output device is headphones or speakers. Specifically, if headphone listening is more sensitive to sound quality degradation than speaker listening, a threshold is used that makes it less likely that culling or integration will be selected so that the sound quality of immersive audio will be higher when listening with headphones than when listening with speakers. Conversely, if speaker listening is more sensitive to sound quality degradation than when listening with headphones, a threshold is used that makes it less likely that culling or integration will be selected so that the sound quality of immersive audio will be higher when listening with speakers than when listening with headphones.

[0303] Furthermore, sounds to be subject to culling or integration may be determined according to the positional relationship between the object and the listener. A threshold related to the hearing characteristics that determines sounds to be subject to culling or integration may be controlled according to the positional relationship between the object and the listener. Specifically, when an object is not visible to the listener (for example, when there is an obstacle between the listener and the object), the threshold related to the hearing characteristics is changed so that the object is more likely to be subject to culling or integration. Conversely, when an object is visible to the listener (for example, when there is no obstacle between the listener and the object), the threshold related to the hearing characteristics is changed so that the object is less likely to be subject to culling or integration. Or the reverse may also be possible.

[0304] Furthermore, sounds to be subject to culling or integration may be determined based on the moving speed of an object. A threshold related to the hearing characteristics that determines sounds to be subject to culling or integration may be controlled based on the moving speed of an object. Specifically, if the moving speed of an object is slow, the threshold related to the hearing characteristics is changed so that the object is more likely to be subject to culling or integration. Conversely, if the moving speed of an object is fast, the threshold related to the hearing characteristics is changed so that the object is less likely to be subject to culling or integration. Or the reverse may be true.

[0305] In the above explanation, direct sound, reflected sound, reverberant sound, and diffracted sound have been used as examples, but the present invention is not limited to these and can be applied to any type of sound, regardless of its name, as long as it is direct sound or a sound derived from direct sound that reaches the listener.

[0306] Furthermore, while the present invention has been described so far based on sound propagation, it is not limited to sound propagation; for example, the present invention can also be applied to light propagation. Regarding light propagation, the present invention is applicable to computer graphics that generate scenes based on direct light, reflected light, and diffracted light. Specifically, in a virtual space or a space that combines virtual space and real space, the light to be culled or integrated is selected based on the relationship between the light reaching the user and the user's visual characteristics. This makes it possible to significantly reduce the amount of calculation required to generate computer graphics while minimizing degradation in the quality of the computer graphics.

[0307] <Functional Description and Modifications of the Rendering Unit> Modifications of the rendering unit described above will be described below with reference to FIGS.

[0308] FIG. 57 is a block diagram of a decoder (rendering unit 5700) according to Modification 1. For the sake of explanation, FIG. 57 shows the direct sound generation unit 1502, reverberation sound generation unit 1503, reflected sound generation unit 1504, and diffracted sound generation unit 1505 arranged in this order, but this order is not necessarily required. Furthermore, the acoustic processing is not limited to this order. Hereinafter, the direct sound generation unit 1502, reverberation sound generation unit 1503, reflected sound generation unit 1504, and diffracted sound generation unit 1505 may be collectively referred to as the sound generation unit. Furthermore, while the figure shows culling units (first culling unit 1506a, second culling unit 1506b, third culling unit 1506c, and fourth culling unit 1506d) arranged before all of the sound generation units, this is merely an example, and any culling unit may be arranged before one or more sound generation units.

[0309] The basic idea of ​​this modified example is that when the number of sounds input to one or more sound generation units exceeds a predetermined value, culling is performed for the number exceeding the predetermined value, so that the number of sounds remains within the predetermined value.

[0310] First, input data (such as a bitstream) is provided to the spatial information management unit 1501. The input data includes an audio signal or encoded audio data representing the audio signal, and metadata used in acoustic processing. If encoded audio data is included, the encoded audio data is provided to an audio data decoder (not shown here), which performs decoding processing to generate an audio signal. This audio signal is provided to the first culling unit 1506a. If an audio signal is included instead of encoded audio data, the audio signal is provided to the first culling unit 1506a. Note that multiple audio signals may be provided to the first culling unit 1506a if there are multiple objects or one object contains multiple sounds, for example.

[0311] The spatial information management unit 1501 extracts metadata from the input data, and the metadata is provided to a direct sound generation unit 1502 , a reverberation sound generation unit 1503 , a reflected sound generation unit 1504 , and a diffracted sound generation unit 1505 .

[0312] The first culling unit 1506a identifies unimportant sounds from the input audio signal, discards the identified sounds, and outputs the remaining sounds to the direct sound generation unit 1502. Note that the signal input to the first culling unit 1506a does not necessarily have to be the input audio signal. For example, it may be another signal not shown here.

[0313] The number of sounds that are left by the first culling unit 1506a is a predetermined value set for the direct sound generation unit 1502, and sounds in a number exceeding the predetermined value are discarded by culling. Sounds that are not discarded by the first culling unit 1506a are output to the direct sound generation unit 1502. If the number of audio signals provided to the first culling unit 1506a is equal to or less than the predetermined value, culling is not performed, and all sounds are output to the direct sound generation unit 1502. Furthermore, this predetermined value may indicate the number of sounds to be culled by the first culling unit 1506a.

[0314] The second culling unit 1506b identifies unimportant sounds from the sounds provided by the direct sound generation unit 1502, discards the identified sounds, and outputs the remaining sounds to the reverberation sound generation unit 1503. Note that the signal input to the second culling unit 1506b does not necessarily have to be the output signal of the direct sound generation unit 1502. For example, it may be an audio signal that is an input signal to the rendering unit 5700, or another signal not shown here.

[0315] The number of sounds to be retained by the second culling unit 1506b is a predetermined value set for the reverberation sound generation unit 1503, and sounds in a number exceeding the predetermined value are discarded by culling. This predetermined value may also indicate the number of sounds to be culled by the second culling unit 1506b. Sounds that are not discarded by the second culling unit 1506b are output to the reverberation sound generation unit 1503. If the number of sounds provided to the second culling unit 1506b is equal to or less than the predetermined value, culling is not performed, and all sounds are output to the reverberation sound generation unit 1503.

[0316] The third culling unit 1506c identifies unimportant sounds from the sounds provided by the reverberation sound generation unit 1503, discards the identified sounds, and outputs the remaining sounds to the reflected sound generation unit 1504. Note that the signal input to the third culling unit 1506c does not necessarily have to be the output signal of the reverberation sound generation unit 1503. For example, it may be an audio signal that is an input signal to the rendering unit 5700, or another signal not shown here.

[0317] The number of sounds to be left by the third culling unit 1506c is a predetermined value set for the reflected sound generation unit 1504, and sounds in a number exceeding the predetermined value are discarded by culling. This predetermined value may also indicate the number of sounds to be culled by the third culling unit 1506c. Sounds that are not discarded by the third culling unit 1506c are output to the reflected sound generation unit 1504. If the number of sounds provided to the third culling unit 1506c is equal to or less than the predetermined value, culling is not performed, and all sounds are output to the reflected sound generation unit 1504.

[0318] The fourth culling unit 1506d identifies unimportant sounds from the sounds provided by the reflected sound generation unit 1504, discards the identified sounds, and outputs the remaining sounds to the diffracted sound generation unit 1505. Note that the signal input to the fourth culling unit 1506d does not necessarily have to be the output signal of the reflected sound generation unit 1504. For example, it may be an audio signal that is an input signal to the rendering unit 5700, or another signal not shown here.

[0319] The number of sounds to be left by the fourth culling unit 1506d is a predetermined value set for the diffracted sound generation unit 1505, and sounds in a number exceeding the predetermined value are discarded by culling. This predetermined value may also indicate the number of sounds to be culled by the fourth culling unit 1506d. Sounds that are not discarded by the fourth culling unit 1506d are output to the diffracted sound generation unit 1505. If the number of sounds provided to the fourth culling unit 1506d is equal to or less than the predetermined value, culling is not performed, and all sounds are output to the diffracted sound generation unit 1505.

[0320] The signals input to the sound generation unit 1507 are the output signals of the respective sound generation units, but they are not limited to these and may be other signals not shown here.

[0321] The predetermined values ​​set for the first culling unit 1506a, the second culling unit 1506b, the third culling unit 1506c, and the fourth culling unit 1506d may be the same value or different values.

[0322] The operation of the rendering unit 5700 according to this modification will be described with reference to Fig. 58. Note that a cross mark in the figure indicates that the sound has been discarded by culling.

[0323] Here, an example is shown in which three audio signals are input. These three audio signals are input to the first culling unit 1506a, and since the predetermined value of the first culling unit 1506a is 2, one signal with the least auditory importance is culled and discarded. The remaining two signals are then provided to the direct sound generation unit 1502.

[0324] The direct sound generation unit 1502 processes the input audio signal to generate direct sound and outputs the direct sound. The direct sound generation unit 1502 here generates one output signal for one input signal. Therefore, this is indicated as "x1" in the figure. If eight output signals are generated for one input signal, the sound generation unit would be indicated as "x8."

[0325] The second culling unit 1506b compares the number of input sounds with a predetermined value, and if the predetermined value exceeds the number of input sounds, culling is performed starting from sounds with the lowest auditory importance by the amount that exceeds the predetermined value. However, in this example, since the number of input sounds is smaller than the predetermined value of the second culling unit 1506b, culling of sounds is not performed, and all input sounds are input to the reverberation sound generation unit 1503.

[0326] The echo generator 1503 performs processing to generate echoes from the two input signals, and outputs 16 echoes.

[0327] The third culling unit 1506c compares the number of input sounds with a predetermined value, and if the predetermined value exceeds the number of input sounds, culling is performed starting from the sounds with the lowest auditory importance by the amount that exceeds the predetermined value. In this example, since the predetermined value is 12 for 16 signals, 4 signals are culled and the remaining 12 signals are output to the reflected sound generation unit 1504.

[0328] The reflected sound generation unit 1504 performs processing to generate reflected sounds from the 12 input signals, and outputs 48 reflected sounds.

[0329] The fourth culling unit 1506d compares the number of input sounds with a predetermined value, and if the predetermined value exceeds the number of input sounds, culling is performed starting from the sounds with the lowest auditory importance by the amount that exceeds the predetermined value. In this example, the predetermined value is 30 for 48 signals, so 18 signals are culled and the remaining 30 are output to the diffracted sound generation unit 1505.

[0330] The diffracted sound generation unit 1505 performs processing to generate diffracted sounds from the 30 input signals, and outputs 60 diffracted sounds.

[0331] The sound generation unit 1507 performs stereophonic processing on the signal provided from the diffracted sound generation unit 1505, and outputs the output signal after the stereophonic processing to the listener. Note that the signals provided to the sound generation unit may be the output signals of the direct sound generation unit 1502, the reverberation sound generation unit 1503, the reflected sound generation unit 1504, and the diffracted sound generation unit 1505, and may further include signals not shown here.

[0332] 59 is a block diagram of a decoder (rendering unit 5900) according to Modification 2. The rendering unit 5900 of this modification differs from the rendering unit 5700 in that a culling unit is arranged after the sound generation unit. In other words, the culling unit may be arranged before the sound generation unit and cull sound signals input to the sound generation unit, or may be arranged after the sound generation unit and cull sound signals output from the sound generation unit.

[0333] The operation of the rendering unit 5900 according to this modification will be described with reference to FIG.

[0334] Here, an example is shown in which three audio signals are input. These three audio signals are input to the direct sound generation unit 1502, which then performs direct sound generation processing on the input audio signals and outputs the direct sound. In the first culling unit 1506a, since the predetermined value of the first culling unit 1506a is 2, one signal with the lowest auditory importance is culled and discarded. The remaining two signals are then provided to the reverberation sound generation unit 1503. Thereafter, sound generation processing and culling processing are performed alternately, as in the first modification example.

[0335] 61 is a block diagram of a decoder (rendering unit 6100) according to Modification 3. The rendering unit 6100 of this modification differs from the rendering unit 5700 in that integrators (first integrator 2001a, second integrator 2001b, third integrator 2001c, and fourth integrator 2001d) are arranged instead of a culling unit. In other words, instead of a culling unit, an integrator may be arranged before the sound generation unit to integrate sound signals input to the sound generation unit.

[0336] The operation of the rendering unit 6100 according to this modification will be described with reference to Fig. 62. Note that the fact that two arrows in the figure are joined together to form a single arrow indicates that a virtual sound has been generated by integrating multiple selected sounds.

[0337] Here, an example is shown in which three audio signals are input. These three audio signals are input to the first integrating unit 2001a. Since the predetermined value of the first integrating unit 2001a is 2, two audio signals with the lowest auditory importance are selected and integrated to generate one virtual sound. This integrated signal and one signal not to be integrated are then provided to the direct sound generating unit 1502. Thereafter, sound generation processing and integration processing are performed alternately, as in the first modification.

[0338] 63 is a block diagram of a decoder (rendering unit 6300) according to Modification 4. The rendering unit 6300 of this modification differs from the rendering unit 6100 in that the integrating unit is arranged after the sound generating unit. In other words, the integrating unit may be arranged before the sound generating unit and integrate the sound signals input to the sound generating unit, or may be arranged after the sound generating unit and integrate the sound signals output from the sound generating unit.

[0339] Another difference is that integrators (first integrator 2001 a, second integrator 2001 b, third integrator 2001 c, and fourth integrator 2001 d) are arranged instead of culling units. In other words, instead of the culling unit, an integrator may be arranged in front of the sound generator and integrate sound signals input to the sound generator.

[0340] The operation of the rendering unit 6300 according to this modification will be described with reference to FIG.

[0341] Here, an example is shown in which three audio signals are input. These three audio signals are input to the direct sound generation unit 1502, which then performs direct sound generation processing on the input audio signals and outputs the direct sound. Since the predetermined value of the first integrating unit 2001a is 2, the first integrating unit 2001a selects two signals with the lowest auditory importance and integrates them to generate one virtual sound. This integrated signal and one signal not to be integrated, totaling two signals, are provided to the reverberation sound generation unit 1503. Thereafter, sound generation processing and integration processing are performed alternately, as in the first modification example.

[0342] FIG. 65 is a block diagram of a decoder (rendering unit 6500) according to the fifth modification.

[0343] The rendering unit 6500 of this modification is characterized in that it sets a small predetermined value used in the culling unit (or integration unit) associated with each sound generation unit when the predetermined value has a large impact on the perceived quality of the sound generated by that sound generation unit, and sets a large predetermined value when the impact is small. This makes it difficult to perform culling or sound integration for sounds for which the listener is likely to perceive deterioration in sound quality, and makes it easier to perform culling or sound integration for sounds for which the listener is unlikely to perceive deterioration in sound quality, thereby reducing the amount of calculation while maintaining the quality of immersive audio.

[0344] The predetermined value setting unit 6501 receives at least one of a control signal, an audio signal, and metadata, sets predetermined values ​​for the first culling unit 1506a to the fourth culling unit 1506d based on that information, and outputs those predetermined values ​​to the corresponding culling units. The first culling unit 1506a to the fourth culling unit 1506d receive those predetermined values ​​and perform culling using those predetermined values.

[0345] In the figure, all of the control signal, audio signal, and metadata are input to the specified value setting unit 6501, but this is for convenience, and in reality, it is sufficient if at least one of the control signal, audio signal, and metadata is input to the specified value setting unit 6501.

[0346] The metadata provided to the predetermined value setting unit 6501 may be information about the indoor environment. The indoor environment is also referred to as an audio scene. For example, if the sound reflection coefficient of walls or obstacles is high, the predetermined value of the culling unit (or integration unit) corresponding to the reflected sound generation unit 1504 and the reverberation sound generation unit 1503 is set to a small value. This makes it difficult for the output signals of the reflected sound generation unit 1504 and the reverberation sound generation unit 1503 to be culled (or integrated), thereby preventing a decrease in the quality of the immersive audio. Furthermore, the influence of such sounds can be reduced by increasing the predetermined value. Furthermore, if there are many obstacles, the predetermined value of the culling unit (or integration unit) corresponding to the diffracted sound generation unit 1505 can be set to a small value, and if there are few obstacles, the predetermined value can be set to a large value. Note that these are merely examples of this embodiment, and the predetermined value of the culling unit (or integration unit) according to the metadata may be controlled by methods other than those exemplified here.

[0347] The control signal provided to the predetermined value setting unit 6501 may be, for example, an instruction from the listener, an instruction from the operator providing the service, or information about the application used by the listener. When a listener or operator wants to emphasize one or more of the direct sound, reverberation sound, reflected sound, and diffracted sound based on their own preferences or ideas, this can be achieved by decreasing the predetermined value of the culling unit (or integrating unit) corresponding to that sound generation unit. Furthermore, when wanting to weaken one or more of the direct sound, reverberation sound, reflected sound, and diffracted sound, this can be achieved by increasing the predetermined value of the culling unit (or integrating unit) corresponding to that sound generation unit. Note that these are merely examples of this embodiment, and the predetermined value of the culling unit (or integrating unit) according to the control signal may be controlled by methods other than those exemplified here.

[0348] The audio signal provided to the predetermined value setting unit 6501 may be used by determining the type of the signal and determining the predetermined value of the culling unit (or integrating unit) corresponding to the sound generation unit based on the determination result. For example, in the case of an audio signal, the predetermined value of the culling unit (or integrating unit) corresponding to the direct sound generation unit may be reduced or the predetermined value of the sound other than the direct sound may be increased to make the content of the speech easier to hear. Furthermore, if the signal provided to the predetermined value setting unit 6501 is an audio signal, the predetermined value of the culling unit (or integrating unit) corresponding to the sound generation unit other than the direct sound may be reduced to enhance the surround effect. Furthermore, if the signal provided to the predetermined value setting unit is a signal emitted by an object, the predetermined value of the culling unit (or integrating unit) corresponding to the direct sound generation unit 1502 may be adjusted depending on the importance of the direction (directivity) of the sound generated from the object reaching the listener. Note that these are merely examples of this embodiment, and the predetermined value of the culling unit (or integrating unit) according to the input signal may be controlled by methods other than those exemplified here.

[0349] 66 and 67 show rendering units 6600, 6700, and 6800, each having a predetermined value setting unit 6501 corresponding to Modifications 2 to 4. The function of the predetermined value setting unit 6501 in each rendering unit is the same as that described above, and therefore description thereof will be omitted here.

[0350] While the present invention has been described using an embodiment in which either the culling unit or the integrating unit is arranged before or after at least one of the multiple sound generation units, this is not limiting, and both the culling unit and the integrating unit may be arranged before or after at least one of the multiple sound generation units. This enables fine control according to the importance of the sound, such as culling sounds with low auditory importance, integrating sounds with medium auditory importance, and not performing any processing on sounds with high auditory importance. This allows for a reduction in the amount of calculation while maintaining the quality of immersive audio.

[0351] Furthermore, in a configuration in which culling is performed first and then sound integration is performed, culling is first performed to reduce the number of sounds that reach the listener, and then sound integration is performed, making it possible to reduce the amount of calculation required for processing to determine the sounds to be integrated, such as which sounds to integrate. Furthermore, in a configuration in which sound integration is first performed and then culling is performed, sounds are first integrated to reduce the number of sounds that reach the listener, and then culling is performed, making it possible to reduce the amount of calculation required for processing to determine the sounds to be culled, such as which sounds to cull. This makes it possible to more effectively reduce the amount of calculation while maintaining the quality of immersive audio.

[0352] Furthermore, a culling unit or integrating unit may be disposed in one of the multiple sound generation units. In the above, an example was shown in which a culling unit or integrating unit is disposed before or after each sound generation unit, but a culling unit or integrating unit does not have to be disposed for each sound generation unit. In other words, it is sufficient if at least one culling unit or integrating unit is disposed in part of the pipeline processing. Furthermore, the culling unit or integrating unit compared the number of sounds input to each sound generation unit or the number of sounds output from each sound generation unit with a predetermined value so that the number of sounds input to each sound generation unit or the number of sounds output from each sound generation unit falls within the predetermined value, but the culling unit or integrating unit does not have to perform the above-described comparison process. In other words, the placement position of the culling or integrating unit may be determined regardless of the number of sounds.

[0353] Furthermore, a culling unit or an integration unit may be arranged before or after one of the multiple sound generation units. In this case, the following effects can be obtained according to the characteristics of each sound generation unit.

[0354] In an audio scene with a strong degree of reverberation relative to the reverberation sound generator, the energy of the reverberation sound relative to the direct sound may become large, making the direct sound difficult to hear. In such cases, by placing a culling unit or an integration unit before or after the reverberation sound generator, the energy of the reverberation sound relative to the direct sound can be reduced, making the direct sound easier to hear.

[0355] If the occurrence of the first or second reflection occurs close in time to the occurrence of the direct sound, the reflected sound may tend to overlap the direct sound, making it difficult to hear the direct sound. In such cases, by placing a culling unit or integrating unit before or after the reflected sound generation unit, the frequency with which reflected sound occurs relative to the direct sound can be reduced, making the direct sound easier to hear.

[0356] In audio scenes with many obstacles in the diffracted sound generator, a lot of diffracted sound is generated. At that time, the energy of the diffracted sound becomes relatively large compared to the direct sound, which can make the direct sound difficult to hear. In such cases, by placing a culling unit or integrating unit before or after the diffracted sound generator, the energy of the diffracted sound can be reduced relative to the direct sound, making the direct sound easier to hear.

[0357] It should be noted that with regard to the direct sound generation unit, it is not common to place a culling unit or integrating unit before or after the direct sound generation unit. In those rare cases, it is necessary to create a special atmosphere by emphasizing the appearance of objects such as people or objects with sounds other than direct sound. The main purpose of placing a culling unit or integrating unit before or after the direct sound generation unit is to achieve the effect of creating such a special atmosphere.

[0358] The culling unit or the integrating unit may operate when certain conditions are met, and may not operate otherwise. Examples of such conditions are shown below.

[0359] For example, the culling unit or the integrating unit may not be operated depending on the audio scene. For example, if the audio scene is outdoors, there are few walls or obstacles that reflect sound, so the number of sounds generated by the reverberation sound generation unit, the reflected sound generation unit, and the diffracted sound generation unit is not very large, and there is little need to reduce the amount of calculation by the culling unit or the integrating unit. By controlling the operation of the culling unit and the integrating unit depending on the audio scene in this way, it is possible to reduce the amount of calculation while avoiding a deterioration in the quality of immersive audio.

[0360] Furthermore, for example, the culling unit or integrating unit may not be operated depending on the type of object. For example, if the object is a human, since human voices generally travel in one direction, the number of sounds generated as reverberation, reflection, or diffraction is not very large. In such cases, there is little need to deploy a culling unit or integrating unit to reduce the amount of calculation. On the other hand, in the case of an object whose generated sound travels in multiple directions (e.g., a car), the number of sounds generated as reverberation, reflection, or diffraction increases, and it is necessary to deploy a culling unit or integrating unit to reduce the amount of calculation. By controlling the operation of the culling unit or integrating unit according to the type of object in this way, it is possible to reduce the amount of calculation while avoiding a decrease in the quality of immersive audio. Furthermore, for example, the culling unit or integrating unit may not be operated depending on the type of target sound. For example, if the target sound is direct sound, reflected sound, or diffracted sound, the characteristics of the object are sufficiently perceived, so the culling unit or integrating unit is not operated to maintain the quality of immersive audio. On the other hand, if the target sound is reverberation, the characteristics of the object are not sufficiently perceived, so the culling unit or integrating unit is operated to reduce the amount of calculation.

[0361] Furthermore, the configuration may be such that the culling unit or integrating unit operates more easily when the types of target sounds are different. For example, when the target sounds are reflected sounds and reverberant sounds, the reflected sounds often make a greater impression on the listener than the reverberant sounds. Therefore, the operation of the culling unit or integrating unit may be controlled so that the sound with the smaller impression is more likely to be culled or two or more sounds including the sound with the smaller impression are more likely to be integrated. The same can be said for other combinations of sound types. On the other hand, when the target sounds are of the same type, each sound needs to be treated equally, so the operation may not be controlled to make culling or sound integration more likely.

[0362] In this way, by controlling the operation of the culling unit or the integration unit depending on the type of target sound, it is possible to reduce the amount of calculation while avoiding degradation of the quality of immersive audio.

[0363] The timing for specifying the operation of the culling unit or the integrating unit may be set to operate when information (a flag) indicating that the culling unit or the integrating unit will operate is received. Examples of the timing for receiving the information (a flag) are shown below.

[0364] For example, whether the culling unit or the integrating unit operates may be determined according to information described in a profile (signaling, configuration information, etc.) at the time of initialization of the acoustic signal processing device. If the information described in the profile (signaling, configuration information, etc.) at the time of initialization corresponds to "operate," the culling unit or the integrating unit operates, and if the information corresponds to "do not operate," the culling unit or the integrating unit does not operate. This eliminates the need for processing to determine whether the culling unit or the integrating unit operates, thereby reducing the amount of calculation.

[0365] Furthermore, for example, whether the culling unit or the integrating unit operates may be determined according to information described in a bitstream received when the audio signal processing device is operating. If the information described in the bitstream corresponds to "operate," the culling unit or the integrating unit operates, and if the information corresponds to "do not operate," the culling unit or the integrating unit does not operate. This eliminates the need for processing to determine whether the culling unit or the integrating unit operates, reducing the amount of calculation. Furthermore, since the determination is made each time a bitstream is received, finer control is possible.

[0366] Furthermore, for example, when an acoustic signal processing device operates using a signal processing thread that performs signal processing and a parameter update thread that updates parameters, it may be possible to determine whether the culling unit or the integrating unit operates at the timing when the parameter update thread operates. Typically, the parameter update thread operates less frequently than the signal processing thread, so it is possible to control whether the culling unit or the integrating unit operates with a small amount of calculation.

[0367] As a variation of the method of setting the predetermined value, the predetermined value of the culling unit (or integration unit) corresponding to the generation unit may be provided by signaling when the audio signal processing device is initialized. As a result, the predetermined value is set at the time of initialization, so that processing for setting the predetermined value while the audio signal processing device is in operation is not required, and an appropriate predetermined value can be used without increasing the amount of calculation.

[0368] The predetermined value of the culling unit (or integration unit) corresponding to the sound generation unit may be provided by metadata while the audio signal processing device is in operation. As a result, the predetermined value is set while the audio signal processing device is in operation, so that even if the importance of each sound generation unit changes over time, it is possible to set a predetermined value appropriate to that importance, and it is possible to always use an appropriate predetermined value.

[0369] While the present invention has been described so far based on sound propagation, it is not limited to sound propagation; the present invention can also be applied to light propagation, for example. Regarding light propagation, the present invention is applicable to computer graphics that generate scenes based on direct light, reflected light, and diffracted light. Specifically, in a virtual space or a space that combines virtual space and real space, the light to be culled or integrated is selected based on the relationship between the light reaching the user and the user's visual characteristics. This makes it possible to significantly reduce the amount of calculation required to generate computer graphics while minimizing degradation in the quality of the computer graphics.

[0370] Other Embodiments Although the embodiments have been described above, the present disclosure is not limited to the above-described embodiments.

[0371] For example, the sound reproduction system described in the above embodiment may be realized as a single device including all of the components, or may be realized by allocating each function to multiple devices and coordinating these multiple devices. In the latter case, the device corresponding to the information processing device may be an information processing device such as a smartphone, tablet terminal, or PC. For example, in the sound reproduction system 100 having a function as a renderer that generates an audio signal with added sound effects, all or part of the renderer's functions may be performed by a server. That is, all or part of the acquisition unit 111, path calculation unit 121, output sound generation unit 131, and signal output unit 141 may be located on a server (not shown). In this case, the sound reproduction system 100 is realized by combining, for example, an information processing device such as a computer or smartphone, a sound presentation device such as a head-mounted display (HMD) or earphones worn by the user 99, and a server (not shown). Note that the computer, sound presentation device, and server may be connected to each other so as to be able to communicate with each other via the same network, or may be connected via different networks. When the sound reproduction system 100 is connected via different networks, the possibility of communication delays increases, so processing on the server may be permitted only when the computer, sound presentation device, and server are connected so as to be able to communicate with each other via the same network. Also, depending on the amount of bitstream data received by the sound reproduction system 100, it may be determined whether the server will take on all or part of the functions of the renderer.

[0372] The sound reproduction system of the present disclosure can also be realized as an information processing device that is connected to a reproduction device having only a driver and that simply reproduces an output sound signal generated based on acquired sound information to the reproduction device. In this case, the information processing device may be realized as hardware having a dedicated circuit, or as software that causes a general-purpose processor to execute specific processing.

[0373] In the above-described embodiment, the processing performed by a specific processing unit may be performed by another processing unit. The order of multiple processing operations may be changed, or multiple processing operations may be performed in parallel.

[0374] In the above-described embodiments, each component may be realized by executing a software program suitable for that component, or by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0375] Furthermore, each component may be realized by hardware. For example, each component may be a circuit (or integrated circuit). These circuits may form a single circuit as a whole, or each may be a separate circuit. Furthermore, each of these circuits may be a general-purpose circuit or a dedicated circuit.

[0376] Furthermore, the general or specific aspects of the present disclosure may be realized as an apparatus, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, etc. Furthermore, the general or specific aspects of the present disclosure may be realized as any combination of an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0377] For example, the present disclosure may be realized as an audio signal reproducing method executed by a computer, or as a program for causing a computer to execute the audio signal reproducing method. The present disclosure may also be realized as a computer-readable non-transitory recording medium on which such a program is recorded.

[0378] In addition, this disclosure also includes forms obtained by applying various modifications to each embodiment that a person skilled in the art would think of, or forms realized by arbitrarily combining the components and functions of each embodiment within the scope that does not deviate from the intent of this disclosure.

[0379] In the present disclosure, the encoded sound information can be rephrased as a bitstream containing a sound signal, which is information about a predetermined sound to be reproduced by the sound reproduction system 100, and metadata, which is information about the localization position when the sound image of the predetermined sound is localized at a predetermined position within a three-dimensional sound field. For example, the sound information may be acquired by the sound reproduction system 100 as a bitstream encoded in a predetermined format such as MPEG-H 3D Audio (ISO / IEC 23008-3). As an example, the encoded sound signal includes information about the predetermined sound to be reproduced by the sound reproduction system 100. The predetermined sound here refers to a sound emitted by a sound source object present in the three-dimensional sound field or a natural environmental sound, and may include, for example, a mechanical sound or the sounds of animals, including humans. When multiple sound source objects are present in the three-dimensional sound field, the sound reproduction system 100 acquires multiple sound signals corresponding to the multiple sound source objects.

[0380] On the other hand, metadata is, for example, information used to control acoustic processing of a sound signal in the sound reproduction system 100. Metadata may be information used to describe a scene expressed in a virtual space (three-dimensional sound field). Here, a scene is a term that refers to a collection of all elements representing three-dimensional images and acoustic events in a virtual space, modeled by the sound reproduction system 100 using metadata. In other words, the metadata here may include not only information for controlling acoustic processing, but also information for controlling video processing. Of course, the metadata may include information for controlling only either audio processing or video processing, or may include information used to control both. In the present disclosure, the bitstream acquired by the sound reproduction system 100 may include such metadata. Alternatively, the sound reproduction system 100 may acquire the metadata separately from the bitstream, as described below.

[0381] The sound reproduction system 100 generates virtual sound effects by performing sound processing on the sound signal using metadata included in the bitstream and additionally acquired position information of the interactive user 99. For example, sound effects such as early reflection sound generation, late reverberation sound generation, diffraction sound generation, distance attenuation effect, localization, sound image localization processing, and Doppler effect may be added. Information for switching on and off all or part of the sound effects may also be added as metadata.

[0382] All or part of the metadata may be obtained from sources other than the bitstream of audio information. For example, either the metadata controlling audio or the metadata controlling video may be obtained from sources other than the bitstream, or both may be obtained from sources other than the bitstream.

[0383] Furthermore, if metadata for controlling the video is included in the bitstream acquired by the audio reproduction system 100, the audio reproduction system 100 may have a function for outputting the metadata that can be used for controlling the video to a display device that displays images or a 3D video reproduction device that reproduces 3D video.

[0384] As an example, the encoded metadata includes information about a three-dimensional sound field including a sound source object emitting a sound and an obstacle object, and information about a localization position when a sound image of the sound is localized at a predetermined position in the three-dimensional sound field (i.e., the sound is perceived as arriving from a predetermined direction), i.e., information about the predetermined direction. Here, an obstacle object is an object that can affect the sound perceived by the user 99 by, for example, blocking or reflecting the sound emitted by the sound source object before it reaches the user 99. Obstacle objects may include not only stationary objects but also animals such as people or moving objects such as machines. Furthermore, when multiple sound source objects exist in a three-dimensional sound field, the other sound source objects may be obstacle objects for any one sound source object. Furthermore, both non-sound-source objects such as building materials or inanimate objects and sound-emitting sound objects may be obstacle objects.

[0385] The spatial information constituting the metadata may include not only the shape of the three-dimensional sound field, but also information representing the shape and position of obstacle objects present in the three-dimensional sound field and the shape and position of sound source objects present in the three-dimensional sound field. The three-dimensional sound field may be either a closed space or an open space, and the metadata may include information representing the reflectance of structures that can reflect sound in the three-dimensional sound field, such as floors, walls, or ceilings, and the reflectance of obstacle objects present in the three-dimensional sound field. Here, the reflectance is the ratio of the energy of reflected sound to incident sound, and is set for each frequency band of sound. Of course, the reflectance may be set uniformly regardless of the frequency band of the sound. Furthermore, when the three-dimensional sound field is an open space, parameters such as a uniform attenuation rate, diffracted sound, or early reflection sound may be used.

[0386] In the above description, reflectance is cited as a parameter related to an obstacle object or a sound source object included in the metadata, but the metadata may include information other than reflectance. For example, information about the material of the object may be included as metadata related to both the sound source object and the non-sound source object. Specifically, the metadata may include parameters such as diffusion rate, transmittance, or sound absorption rate.

[0387] Information about the sound source object may include information such as volume, radiation characteristics (directivity), playback conditions, the number and type of sound sources emitted from a single object, or information specifying a sound source area within the object. The playback conditions may, for example, determine whether the sound is a continuous sound or an event-triggering sound. The sound source area within the object may be determined based on the relative relationship between the position of the user 99 and the position of the object, or may be determined based on the object. When the sound source area is determined based on the relative relationship between the position of the user 99 and the position of the object, the surface from which the user 99 is viewing the object is used as the reference, and the user 99 can be made to perceive sound X as emanating from the right side of the object and sound Y as emanating from the left side of the object as viewed from the user 99. When the sound source area is determined based on the object, the surface from which the user 99 is viewing the object is used as the reference, and the sound emitted from which area of ​​the object can be fixed regardless of the direction the user 99 is viewing. For example, the user 99 can be made to perceive a high-pitched sound coming from the right side and a low-pitched sound coming from the left side when viewing the object from the front. In this case, when the user 99 goes around to the back of the object, the user 99 can be made to perceive a low-pitched sound coming from the right side and a high-pitched sound coming from the left side as viewed from the back.

[0388] The spatial metadata may include the time to early reflections, the reverberation time, or the ratio of direct sound to diffuse sound. If the ratio of direct sound to diffuse sound is zero, the user 99 will perceive only direct sound.

[0389] Furthermore, information indicating the position and orientation of the user 99 in the three-dimensional sound field may be included in the bitstream as metadata as an initial setting, or may not be included in the bitstream. If the information indicating the position and orientation of the user 99 is not included in the bitstream, the information indicating the position and orientation of the user 99 is acquired from information other than the bitstream. For example, the position information of the user 99 in the VR space may be acquired from an app that provides VR content. The position information of the user 99 for presenting sound as AR may be position information obtained by a mobile terminal performing self-position estimation using GPS, a camera, LiDAR (Laser Imaging Detection and Ranging), or the like. Note that the sound signal and metadata may be stored in a single bitstream or may be stored separately in multiple bitstreams. Similarly, the sound signal and metadata may be stored in a single file or may be stored separately in multiple files.

[0390] When an audio signal and metadata are stored separately in multiple bitstreams, information indicating other related bitstreams may be included in one or some of the multiple bitstreams in which the audio signal and metadata are stored. Also, information indicating other related bitstreams may be included in metadata or control information of each of the multiple bitstreams in which the audio signal and metadata are stored. When an audio signal and metadata are stored separately in multiple files, information indicating other related bitstreams or files may be included in one or some of the multiple files in which the audio signal and metadata are stored. Also, information indicating other related bitstreams or files may be included in metadata or control information of each of the multiple bitstreams in which the audio signal and metadata are stored.

[0391] Here, the related bitstreams or files are, for example, bitstreams or files that may be used simultaneously during audio processing. Furthermore, information indicating other related bitstreams may be collectively described in the metadata or control information of one bitstream among multiple bitstreams storing audio signals and metadata, or may be separately described in the metadata or control information of two or more bitstreams among the multiple bitstreams storing audio signals and metadata. Similarly, information indicating other related bitstreams or files may be collectively described in the metadata or control information of one file among multiple files storing audio signals and metadata, or may be separately described in the metadata or control information of two or more files among the multiple files storing audio signals and metadata. Furthermore, a control file collectively describing information indicating other related bitstreams or files may be generated separately from the multiple files storing audio signals and metadata. In this case, the control file does not need to store the audio signal and metadata.

[0392] Here, the information indicating the other related bitstream or file may be, for example, an identifier indicating the other bitstream, a file name indicating the other file, a URL (Uniform Resource Locator), or a URI (Uniform Resource Identifier). In this case, the acquisition unit identifies or acquires the bitstream or file based on the information indicating the other related bitstream or file. Furthermore, the information indicating the other related bitstream may be included in metadata or control information of at least some of the bitstreams among a plurality of bitstreams storing audio signals and metadata, and the information indicating the other related file may be included in metadata or control information of at least some of the files among a plurality of files storing audio signals and metadata. Here, the file including information indicating the related bitstream or file may be, for example, a control file such as a manifest file used for content distribution.

[0393] The present disclosure is useful in reproducing sound, for example, by allowing a user to perceive stereoscopic sound.

[0394] 99 User 100 Sound reproduction system 101 Information processing device 102 Communication module 103 Detector 104 Driver 105 Database 111 Acquisition unit 112 Encoded sound information input unit 113 Decoding processing unit 114 Sensing information input unit 115 Characteristics acquisition unit 121 Path calculation unit 131 Output sound generation unit 132 Reduction processing unit 133 Culling unit 134 Integration unit 141 Signal output unit 300 3D video reproduction device

Claims

1. An audio processing device comprising: an acquisition unit that acquires audio information including an audio signal and information on the position of a sound source object within a three-dimensional sound field; a characteristic acquisition unit that acquires information on a user's hearing characteristics; and a reduction processing unit that, when generating an output sound signal from the audio signal included in the acquired sound information, reduces at least one sound signal based on the acquired information on the user's hearing characteristics, thereby generating the output sound signal that does not include the signal.

2. The sound processing device according to claim 1, wherein the information regarding the hearing characteristics of the user is information regarding whether or not the user is able to distinguish between two or more sounds arriving toward the user.

3. The sound processing device described in claim 2, wherein the information relating to the user's hearing characteristics includes information regarding angles of two or more sounds arriving toward the user, and the reduction processing unit reduces the signal of at least one of the two or more sounds based on the angle indicated by the information included in the information relating to the user's hearing characteristics.

4. The sound processing device described in claim 2, wherein the information regarding the user's hearing characteristics includes information regarding a distance difference between two or more sounds arriving toward the user, and the reduction processing unit reduces the signal of at least one of the two or more sounds based on the distance difference indicated by the information included in the information regarding the user's hearing characteristics.

5. The sound processing device according to claim 2, wherein the information relating to the user's hearing characteristics includes information regarding a level ratio of two or more sounds arriving toward the user, and the reduction processing unit reduces the signal of at least one of the two or more sounds based on the level ratio indicated by the information included in the information relating to the user's hearing characteristics.

6. The audio processing device according to claim 2, wherein the information relating to the user's hearing characteristics includes information regarding a signal energy ratio of two or more sounds arriving toward the user, and the reduction processing unit reduces the signal of at least one of the two or more sounds based on the signal energy ratio indicated by the information included in the information relating to the user's hearing characteristics.

7. The sound processing device of claim 2, wherein the information relating to the user's hearing characteristics includes information regarding an angle and a level ratio of two or more sounds arriving toward the user, and the reduction processing unit reduces the signal of at least one of the two or more sounds based on the angle and the level ratio indicated by the information included in the information relating to the user's hearing characteristics.

8. The sound processing device of claim 2, wherein the information relating to the user's hearing characteristics includes information regarding an angle and a signal energy ratio of two or more sounds arriving toward the user, and the reduction processing unit reduces the signal of at least one of the two or more sounds based on the angle and the signal energy ratio indicated by the information included in the information relating to the user's hearing characteristics.

9. An audio processing device as described in any one of claims 1 to 8, wherein the information relating to the user's hearing characteristics includes information regarding the level of sensitivity for each direction of sound coming toward the user, and the reduction processing unit preferentially reduces sounds from directions with low sensitivity over sounds from directions with high sensitivity based on the level of sensitivity indicated by the information included in the information relating to the user's hearing characteristics.

10. The sound processing device according to claim 9, wherein the sensitivity is higher the closer to the front of the user, and lower the closer to the back of the user.

11. The sound processing device according to claim 9, wherein the high and low sensitivity includes a 360° vertical sensitivity distribution of the user and a 360° horizontal sensitivity distribution of the user.

12. The sound processing device according to claim 11, wherein the distribution of sensitivity in the horizontal direction of 360° is finer than the distribution of sensitivity in the vertical direction of 360°.

13. The sound processing device according to claim 1, wherein the reduction processing unit includes a culling unit that reduces at least one sound signal by discarding the at least one sound signal.

14. The sound processing device according to claim 1, wherein the reduction processing unit includes an integration unit that reduces the two sound signals by discarding at least two sound signals and supplementing the at least two sound signals with a single virtual sound signal obtained by integrating the two sound signals.

15. The sound processing device according to claim 1, wherein the reduction processing unit includes: a culling unit that discards at least one sound signal and reduces the at least one sound signal; and an integration unit that discards at least two sound signals and reduces the at least two sound signals by supplementing a single virtual sound signal obtained by integrating the at least two sound signals.

16. The sound processing device according to claim 1, wherein the reduction processing unit reduces at least one sound signal based on the acquired information on the hearing characteristics of the user and the type of sound.

17. The sound processing device according to claim 14 or 15, wherein the integration unit generates the virtual sound signal by discarding at least two sound signals and adding up the at least two sound signals.

18. The sound processing device according to claim 17, wherein the integration unit discards at least two sound signals, adjusts at least one of a phase and an energy of at least one of the at least two sound signals, and generates the virtual sound signal by adding the at least two sound signals after adjustment.

19. The sound processing device according to claim 1, wherein the reduction processing unit performs the reduction of at least one sound signal gradually in the time domain.

20. The sound processing device according to claim 1, wherein the reduction processing unit performs at least one of the following: discarding at least one sound signal input to at least one of the processes for generating each of a plurality of sound signals from the acoustic signal before the process; and discarding at least one sound signal generated in at least one of the processes for generating each of a plurality of sound signals from the acoustic signal after the process.

21. The sound processing device according to claim 1, wherein the reduction processing unit performs at least one of the following: discarding at least one sound signal input to a process for generating each of a plurality of sound signals from the acoustic signal before the process for generating at least a diffracted sound; and discarding at least one diffracted sound signal generated in the process for generating each of a plurality of sound signals from the acoustic signal after the process for generating at least a diffracted sound.

22. An acoustic processing method executed by a computer, comprising: a step of acquiring sound information including an acoustic signal and information on the position of a sound source object in a three-dimensional sound field; a step of acquiring information on a user's hearing characteristics; and a step of generating an output sound signal from the acoustic signal included in the acquired sound information, reducing at least one sound signal based on the acquired information on the user's hearing characteristics, thereby generating the output sound signal that does not include the signal.

23. A program for causing the computer to execute the acoustic processing method according to claim 22.

Images