Acoustic processing apparatus, acoustic processing method, and program

Abstract

An acoustic processing apparatus (information processing apparatus (101)) comprises: an acquisition unit (111) that acquires sound information including an acoustic signal and information on the position of a sound source object in a three-dimensional sound field; a table reference unit (116) that refers to a table that associates at least one of the position of a user (99) and the position of the sound source object in the three-dimensional sound field with information for determining a target sound; and a reduction processing unit (132) that determines the target sound using the information for determining the target sound obtained by referring to the table, and generates an output sound signal that does not include the signal of the determined target sound by reducing the signal of the determined target sound from among a plurality of sound signals generated when generating the output sound signal from the acoustic signal included in the sound information acquired.

Description

Sound processing device, sound processing method, and program

[0001] The present disclosure relates to a sound processing device, a sound processing method, and a program.

[0002] Conventionally, there are known techniques for reproducing sound in a virtual three-dimensional space to allow a user to perceive stereoscopic sound (see, for example, Patent Document 1). Furthermore, in order to perceive sound as if it is coming from a sound source object to the user in such a three-dimensional space, processing is required to generate output sound information from original sound information. In particular, reproducing stereoscopic sound in response to the user's body movements in a virtual space requires extensive processing. Advances in computer graphics (CG) have made it relatively easy to create visually complex virtual environments, making technology for realizing corresponding auditory information important. Additionally, when processing from sound information to output sound information is performed in advance, a large memory area is required to store the pre-calculated processing results. Furthermore, transmitting such large amounts of processed data may require a wide communication bandwidth.

[0003] To achieve a more realistic sound environment, the number of objects that emit sound in the virtual three-dimensional space must be increased, and secondary sounds based on acoustic effects such as reflected sound, diffracted sound, and reverberation must be increased. Furthermore, these secondary sounds must be appropriately changed in response to the user's movements, which requires a large amount of processing power.

[0004] Japanese Patent Application Laid-Open No. 2020-18620

[0005] Therefore, an object of the present disclosure is to provide a sound processing device or the like that can generate an output sound signal appropriately in terms of the amount of processing.

[0006] A sound processing device according to one aspect of the present disclosure includes an acquisition unit that acquires sound information including an acoustic signal and information on the position of a sound source object within a three-dimensional sound field; a table reference unit that references a table that associates at least one of the position of a user and the position of the sound source object within the three-dimensional sound field with information for determining a target sound; and a reduction processing unit that determines the target sound using the information for determining the target sound obtained by reference to the table, and reduces the signal of the determined target sound from among multiple sound signals generated when generating an output sound signal from the acoustic signal included in the acquired sound information, thereby generating an output sound signal that does not include the signal.

[0007] Furthermore, an acoustic processing method according to one aspect of the present disclosure is an acoustic processing method executed by a computer, and includes the steps of: acquiring sound information including an acoustic signal and information on the position of a sound source object within a three-dimensional sound field; referring to a table that associates at least one of the user's position and the sound source object's position within the three-dimensional sound field with information for determining a target sound; determining the target sound using the information for determining the target sound obtained by referring to the table; and generating an output sound signal that does not include the determined target sound by reducing the signal for the determined target sound from multiple sound signals generated when generating an output sound signal from the acoustic signal included in the acquired sound information.

[0008] Furthermore, one aspect of the present disclosure can also be realized as a program for causing a computer to execute the above-described acoustic processing method.

[0009] These comprehensive or specific aspects may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0010] According to the present disclosure, it is possible to appropriately generate an output sound signal.

[0011] FIG. 1 is a schematic diagram showing a use example of a sound reproduction system according to an embodiment. FIG. 2 is a block diagram showing a functional configuration of the sound reproduction system according to an embodiment. FIG. 3 is a diagram for explaining an example of an audio signal according to an embodiment. FIG. 4 is a block diagram showing a functional configuration of an acquisition unit according to an embodiment. FIG. 5 is a block diagram showing a functional configuration of an output sound generation unit according to an embodiment. FIG. 6 is a diagram for explaining another example of a sound reproduction system according to an embodiment. FIG. 7 is a diagram for explaining another example of a sound reproduction system according to an embodiment. FIG. 8 is a diagram for explaining another example of a sound reproduction system according to an embodiment. FIG. 9 is a diagram for explaining another example of a sound reproduction system according to an embodiment. FIG. 10 is a diagram for explaining another example of a sound reproduction system according to an embodiment. FIG. 11 is a diagram for explaining another example of a sound reproduction system according to an embodiment. FIG. 12 is a diagram for explaining another example of a sound reproduction system according to an embodiment. FIG. 13 is a diagram for explaining another example of a sound reproduction system according to an embodiment. FIG. 14 is a diagram for explaining another example of a sound reproduction system according to an embodiment. FIG. 15 is a diagram for explaining a specific example of a sound reproduction system according to Example 1 of an embodiment. FIG. 16 is a diagram for explaining a specific example of a sound reproduction system according to Example 1 of an embodiment. FIG. 17 is a diagram for explaining a specific example of the sound reproduction system according to Example 1 of the embodiment. FIG. 18 is a diagram for explaining a specific example of the sound reproduction system according to Example 1 of the embodiment. FIG. 19 is a diagram for explaining a specific example of the sound reproduction system according to Example 1 of the embodiment. FIG. 20 is a diagram for explaining a specific example of the sound reproduction system according to Example 1 of the embodiment. FIG. 21 is a diagram for explaining a specific example of the sound reproduction system according to Example 1 of the embodiment. FIG. 22 is a diagram for explaining a specific example of the sound reproduction system according to Example 1 of the embodiment. FIG. 23 is a diagram for explaining a specific example of the sound reproduction system according to Example 1 of the embodiment. FIG. 24 is a diagram for explaining a specific example of the sound reproduction system according to Example 1 of the embodiment. FIG. 25 is a diagram for explaining a specific example of the sound reproduction system according to Example 1 of the embodiment.FIG. 26 is a diagram for explaining a specific example of a sound reproduction system according to Example 1 of the embodiment. FIG. 27 is a diagram for explaining a specific example of a sound reproduction system according to Example 1 of the embodiment. FIG. 28 is a diagram for explaining a specific example of a sound reproduction system according to Example 1 of the embodiment. FIG. 29 is a diagram for explaining a specific example of a sound reproduction system according to Example 1 of the embodiment. FIG. 30 is a diagram for explaining a specific example of a sound reproduction system according to Example 1 of the embodiment. FIG. 31 is a diagram for explaining a specific example of a sound reproduction system according to Example 1 of the embodiment. FIG. 32 is a diagram for explaining a specific example of a sound reproduction system according to Example 1 of the embodiment. FIG. 33 is a diagram for explaining a specific example of a sound reproduction system according to Example 1 of the embodiment. FIG. 34 is a diagram for explaining a specific example of a sound reproduction system according to Example 1 of the embodiment. FIG. 35 is a diagram for explaining a specific example of a sound reproduction system according to Example 2 of the embodiment. FIG. 36 is a diagram for explaining a specific example of a sound reproduction system according to Example 2 of the embodiment. FIG. 37 is a diagram for explaining a specific example of a sound reproduction system according to Example 2 of the embodiment. FIG. 38 is a diagram for explaining a specific example of a sound reproduction system according to Example 2 of the embodiment. FIG. 39 is a diagram for explaining a specific example of a sound reproduction system according to Example 2 of the embodiment. FIG. 40 is a diagram for explaining a specific example of a sound reproduction system according to Example 2 of the embodiment. FIG. 41 is a diagram for explaining a specific example of a sound reproduction system according to Example 2 of the embodiment. FIG. 42 is a diagram for explaining a specific example of a sound reproduction system according to Example 2 of the embodiment. FIG. 43 is a diagram for explaining a specific example of a sound reproduction system according to Example 2 of the embodiment. FIG. 44 is a diagram for explaining a specific example of a sound reproduction system according to Example 2 of the embodiment. FIG. 45 is a diagram for explaining a specific example of a sound reproduction system according to Example 2 of the embodiment. FIG. 46 is a diagram for explaining a specific example of a sound reproduction system according to Example 2 of the embodiment. FIG. 47 is a diagram for explaining a specific example of a sound reproduction system according to Example 2 of the embodiment. FIG. 48 is a diagram for explaining a specific example of a sound reproduction system according to Example 2 of the embodiment.FIG. 49 is a diagram for explaining a specific example of a sound reproduction system according to Example 2 of the embodiment. FIG. 50 is a diagram for explaining a specific example of a sound reproduction system according to Example 2 of the embodiment. FIG. 51 is a diagram for explaining a specific example of a sound reproduction system according to Example 2 of the embodiment. FIG. 52 is a diagram for explaining a specific example of a sound reproduction system according to Example 2 of the embodiment. FIG. 53 is a diagram for explaining a specific example of a sound reproduction system according to Example 2 of the embodiment. FIG. 54 is a diagram for explaining a specific example of a sound reproduction system according to Example 2 of the embodiment. FIG. 55 is a diagram for explaining a specific example of a sound reproduction system according to Example 2 of the embodiment. FIG. 56 is a diagram for explaining a specific example of a sound reproduction system according to Example 2 of the embodiment. FIG. 57 is a diagram for explaining a specific example of a sound reproduction system according to Example 2 of the embodiment. FIG. 58 is a diagram for explaining a specific example of a sound reproduction system according to Example 3 of the embodiment. FIG. 59 is a diagram for explaining a specific example of a sound reproduction system according to Example 3 of the embodiment. FIG. 60 is a diagram for explaining a specific example of a sound reproduction system according to Example 3 of the embodiment. FIG. 61 is a diagram for explaining a specific example of a sound reproduction system according to Example 3 of the embodiment. FIG. 62A is a diagram for explaining a specific example of a sound reproduction system according to Example 3 of an embodiment. FIG. 62B is a diagram for explaining a specific example of a sound reproduction system according to Example 3 of an embodiment. FIG. 63 is a diagram for explaining a specific example of a sound reproduction system according to Example 3 of an embodiment. FIG. 64 is a diagram for explaining a specific example of a sound reproduction system according to Example 3 of an embodiment. FIG. 65 is a diagram for explaining a specific example of a sound reproduction system according to Example 3 of an embodiment. FIG. 66 is a diagram for explaining a specific example of a sound reproduction system according to Example 3 of an embodiment. FIG. 67 is a diagram for explaining a specific example of a sound reproduction system according to Example 3 of an embodiment. FIG. 68 is a diagram for explaining a specific example of a sound reproduction system according to Example 3 of an embodiment. FIG. 69 is a diagram for explaining a specific example of a sound reproduction system according to Example 3 of an embodiment. FIG. 70 is a diagram for explaining a specific example of a sound reproduction system according to Example 3 of an embodiment.

[0012] (Knowledge that forms the basis of the disclosure) Conventionally, a technology related to sound reproduction that allows a user to perceive stereoscopic sound in a virtual three-dimensional space (hereinafter sometimes referred to as a three-dimensional sound field) has been known (see, for example, Patent Document 1). Using this technology, a user can perceive sound as if a sound source object exists at a predetermined position in the virtual space and the sound is coming from that direction. In order to localize a sound image at a predetermined position in the virtual three-dimensional space in this way, for example, calculation processing is required for a sound signal emitted by a sound source object (also referred to as a sound emitted from the sound source object or a reproduced sound) to generate a sound arrival time difference between the two ears and a sound level difference (or sound pressure difference) between the two ears that causes the sound to be perceived as stereoscopic sound. Such calculation processing is performed by applying a stereophonic filter. A stereophonic filter is an information processing filter that, when an output sound signal obtained by applying the filter to original sound information is reproduced, causes the position (such as the direction and distance) of the sound, the size of the sound source, the width of the space, and the like to be perceived with a three-dimensional effect.

[0013] As an example of the computational process for applying such a stereophonic filter, a process is known in which a head-related transfer function (HRTF) is convolved with a target sound signal to make the sound perceived as coming from a predetermined direction. By performing this HRTF convolution process at a sufficiently fine angle with respect to the direction of arrival of the reproduced sound from the position of the sound source object to the user's position, the sense of realism experienced by the user is improved.

[0014] In recent years, there has been active development of technologies related to virtual reality (VR). Virtual reality focuses on appropriately changing the position of a sound source object in a virtual three-dimensional space in response to a user's movements, allowing the user to experience the sensation of moving within the virtual space. To achieve this, it is necessary to move the localization position of a sound image in the virtual space relative to the user's movements. This processing has been performed by applying a stereophonic filter, such as the head-related transfer function described above, to the original sound information. However, when a user moves within a three-dimensional space, the sound transmission path changes from moment to moment due to changes in the positional relationship between the sound source object and the user, such as due to sound reverberation and interference. This requires determining the sound transmission path from the sound source object based on the positional relationship between the sound source object and the user, and convolving the transfer function to take into account sound reverberation and interference. However, such information processing requires a huge amount of processing, and an improvement in the sense of realism may not be achieved without a large-scale processing device.

[0015] Therefore, in order to reduce such an enormous amount of processing, attempts have been made to reduce a portion of the sounds being reproduced. Specifically, for each of the multiple sound source objects in a three-dimensional space, or for each of the multiple types of sounds generated from each of the sound source objects, rather than convolving a head-related transfer function with each of them, a portion of the sounds is reduced and then the head-related transfer function is convolved. By doing so, in the convolution of the head-related transfer function, which requires a particularly large amount of processing, that is, in the process of generating a stereophonic signal for output (in other words, an output signal or an output sound signal), the number of sound signals to be processed is reduced, and a significant reduction in the processing amount is expected.

[0016] However, since indiscriminate reduction of sound signals will result in sound degradation, a table of sounds that will cause relatively little sound degradation is prepared in advance to suppress this sound degradation, and the sounds to be reduced are determined by referring to this table. This makes it possible to realize a sound processing device that can suppress sound degradation while reducing the amount of processing.

[0017] A more specific outline of the present disclosure is as follows.

[0018] A sound processing device according to a first aspect of the present disclosure includes an acquisition unit that acquires sound information including an acoustic signal and information on the position of a sound source object within a three-dimensional sound field; a table reference unit that references a table that associates at least one of the position of a user and the position of the sound source object within the three-dimensional sound field with information for determining a target sound; and a reduction processing unit that determines the target sound using the information for determining the target sound obtained by referencing the table, and generates an output sound signal that does not include the signal of the determined target sound by reducing the signal of the determined target sound from multiple sound signals that are generated when generating an output sound signal from the acoustic signal included in the acquired sound information.

[0019] In other words, the sound processing device according to the first aspect is a sound processing device that refers to a table prepared in advance based on at least one of the position of the object and the position of the listener among multiple sounds that emanate from an object (sound source object) and reach a listener (user) via multiple paths, and outputs the contents shown in the table to subsequent processing (reduction processing).

[0020] According to such a sound processing device, a table of sounds that have relatively little sound degradation for at least one of the user's position and the sound source object's position in a three-dimensional sound field is prepared in advance, and the sounds to be reduced are determined by referring to the table. This makes it possible to suppress sound degradation while reducing the amount of processing. Therefore, it is possible to generate an output sound signal appropriately in terms of the amount of processing.

[0021] In addition, a sound processing device according to a second aspect is the sound processing device according to the first aspect, wherein the table includes a priority order for each of a plurality of sounds, with the lower the priority order, the more likely the sound is to be determined as the target sound.

[0022] That is, the sound processing device according to the second aspect is a sound processing device in which the table indicates the priority of sounds that reach the listener.

[0023] According to such a sound processing device, the table can contain the priority of each of a plurality of sounds, with the lower the priority, the more likely the sound is to be determined as the target sound, and the sound to be reduced can be determined by referring to the table.

[0024] Furthermore, a sound processing device according to a third aspect is the sound processing device according to the first aspect, wherein the table includes an identifier for distinguishing each of the sounds determined as at least two target sounds from the other sounds among the plurality of sounds.

[0025] That is, the sound processing device according to the third aspect is a sound processing device in which the table indicates identifiers of two or more sounds that reach the listener.

[0026] According to such a sound processing device, the table can include identifiers for distinguishing at least two of the multiple sounds determined as target sounds from the other sounds, and the sounds to be reduced can be determined by referring to the table.

[0027] A sound processing device according to a fourth aspect is the sound processing device according to the first aspect, wherein the table includes information indicating whether or not to perform an operation to generate each of a plurality of sounds.

[0028] That is, the sound processing device according to the fourth aspect is a sound processing device in which the table indicates operation / non-operation information of the sound that reaches the listener.

[0029] According to such a sound processing device, the table can contain information indicating whether or not to perform an operation to generate each of a plurality of sounds, and the sounds to be reduced can be determined by referring to the table.

[0030] Furthermore, a sound processing device according to a fifth aspect is the sound processing device according to the first aspect, in which the table includes information indicating whether or not to perform an operation to generate each of a plurality of sounds, and a priority of each of the plurality of sounds, where the lower the priority, the more likely the sound is to be determined as the target sound.

[0031] That is, the sound processing device according to the fifth aspect is a sound processing device in which the table includes both the operation / non-operation information and the priority of the sounds that reach the listener.

[0032] According to such a sound processing device, the table can contain information indicating whether or not to operate to generate each of a plurality of sounds, and a priority order for each of the plurality of sounds, where the lower the priority, the more likely the sound is to be determined as the target sound, and the sound to be reduced can be determined by referring to the table.

[0033] Furthermore, a sound processing device according to a sixth aspect is a sound processing device according to any one of the first to fifth aspects, in which the reduction processing unit includes a culling unit that reduces the signal of the target sound by discarding the signal of the target sound.

[0034] That is, the sound processing device according to the sixth aspect is a sound processing device in which the subsequent processing is to cull the sounds shown in the table.

[0035] According to such a sound processing device, the signal of the determined target sound can be reduced by discarding the signal of the target sound.

[0036] Furthermore, a sound processing device according to a seventh aspect is the sound processing device according to the sixth aspect, in which the reduction processing unit discards the target sound signal by stopping the operation of generating the target sound signal.

[0037] That is, the sound processing device according to the seventh aspect is a sound processing device in which the subsequent processing stops the operation of the sound generation unit shown in the table.

[0038] According to such a sound processing device, the operation of generating a target sound signal is stopped, and the target sound signal is discarded, thereby reducing the determined target sound signal.

[0039] Furthermore, a sound processing device according to an eighth aspect is a sound processing device according to any one of the first to fifth aspects, in which the reduction processing unit includes an integration unit that reduces the signals of at least two target sounds by discarding the signals of at least two target sounds and supplementing them with a single virtual sound signal that is obtained by integrating the signals of the at least two target sounds.

[0040] That is, the sound processing device according to the eighth aspect is a sound processing device in which the subsequent processing integrates the sounds shown in the table into a number of sounds that is smaller than the number of sounds shown in the table.

[0041] According to such an audio processing device, the sound signals of at least two targets can be reduced by discarding the sound signals of the at least two targets and supplementing them with a single virtual sound signal that integrates the sound signals of the at least two targets.

[0042] Furthermore, a sound processing device according to a ninth aspect is the sound processing device according to the eighth aspect, in which the integration unit supplements the virtual sound signal so that the virtual sound is localized at a position based on the positions of the at least two discarded target sounds and the position of the user.

[0043] That is, the sound processing device according to the ninth aspect is a sound processing device in which the sound generated after integration is emitted from a position within an area determined based on the positions of the sounds to be integrated and the position of the listener.

[0044] According to such a sound processing device, it is possible to compensate for the virtual sound signal so that the virtual sound is localized at a position based on the positions of the at least two discarded target sounds and the position of the user.

[0045] Furthermore, a sound processing device according to a tenth aspect is the sound processing device according to the third aspect, wherein the table is pre-created to include identifiers of at least two sounds determined as at least two target sounds, based on at least one of information regarding the angles of two or more sounds arriving toward the user and information regarding the level ratios of two or more sounds arriving toward the user.

[0046] In other words, the sound processing device of the tenth aspect is a sound processing device in which the sound to be integrated is determined based on the intersection angle (angle formed) of the two sounds when they reach the listener and the level ratio of the two sounds.

[0047] With such a sound processing device, the sounds to be reduced can be determined by referring to a table created in advance to include identifiers of at least two sounds based on at least one of information regarding the angle of two or more sounds arriving toward the user and information regarding the level ratio of two or more sounds arriving toward the user, as sounds to be determined as at least two target sounds.

[0048] Furthermore, an eleventh aspect of the present invention is a sound processing device according to any one of the first to tenth aspects, wherein the reduction processing unit gradually reduces the signal of at least one sound in the time domain.

[0049] In other words, the sound processing device of the eleventh aspect is a sound processing device in which the predetermined processing (later processing) includes processing in which, when the identifier of the sound to be integrated changes over time, the integration processing smoothly transitions from before the change to after the change.

[0050] According to such a sound processing device, it is possible to gradually reduce the sound by reducing at least one sound signal in the time domain.

[0051] Furthermore, a sound processing device according to a twelfth aspect is a sound processing device according to any one of the first to eleventh aspects, in which the reduction processing unit corrects information for determining the target sound obtained by referring to a table, and determines the target sound using the corrected information for determining the target sound.

[0052] That is, the sound processing device according to the twelfth aspect is a sound processing device in which the subsequent processing generates a sound that reaches the listener based on the content obtained by referencing the table.

[0053] With such a sound processing device, the information for determining the target sound obtained by referring to the table can be corrected, and the target sound can be determined using the corrected information for determining the more appropriate target sound.

[0054] Furthermore, a sound processing device according to a thirteenth aspect is a sound processing device according to any one of the first to twelfth aspects, in which the table reference unit converts each of the user's position and the sound source object's position using the positions of lattice points that divide the three-dimensional sound field into unit spaces of a predetermined size, and references the table using at least one of the converted user's position and sound source object's position.

[0055] That is, the sound processing device according to the thirteenth aspect is a sound processing device in which the position of an object and the position of a listener are represented by the positions of lattice points set in space at an arbitrary resolution.

[0056] With such an audio processing device, the user's position and the sound source object's position can each be converted using the positions of lattice points that divide the three-dimensional sound field into unit spaces of a predetermined size, and the table can be referenced using at least one of the converted user's position and sound source object's position.

[0057] In addition, a sound processing device according to a 14th aspect is a sound processing device according to any one of the first to 12th aspects, in which the user's position and the sound source object's position are each converted using an identifier of a voxel, which is a unit space that divides a three-dimensional sound field into multiple sizes, and the table is referenced using at least one of the converted user's position and sound source object's position.

[0058] That is, the sound processing device according to the fourteenth aspect is a sound processing device in which the position of an object and the position of a listener are expressed in space by identifiers of voxels of any size.

[0059] With such an audio processing device, the user's position and the sound source object's position can each be converted using an identifier of a voxel, which is a unit space that divides a three-dimensional sound field into multiple sizes, and the table can be referenced using at least one of the converted user's position and sound source object's position.

[0060] A sound processing device according to a fifteenth aspect is the sound processing device according to the thirteenth or fourteenth aspect, in which the size of the unit space is changed based on at least one of the storage capacity and the processing capacity of the sound processing device.

[0061] That is, the sound processing device according to the fifteenth aspect is a sound processing device in which the interval between lattice points or the size of a voxel is determined based on the storage capacity and processing capability of the device in which the sound processing device is implemented.

[0062] With such a sound processing device, the size of the unit space is set based on at least one of the memory capacity and processing power of the sound processing device, and the table can be referenced using at least one of the user's position and the sound source object's position converted using the lattice point position or the voxel identifier.

[0063] In addition, a sound processing device according to a sixteenth aspect is a sound processing device according to any one of the first to fifteenth aspects, and further includes a table setting unit that sets a table when the sound processing device is initialized.

[0064] That is, the sound processing device according to the sixteenth aspect is a sound processing device in which the table is set when the sound processing device is initialized.

[0065] According to such a sound processing device, the table set when the sound processing device is initialized can be used in subsequent processes.

[0066] A sound processing device according to a seventeenth aspect is the sound processing device according to the sixteenth aspect, wherein the table setting unit further updates at least a part of the table when information for updating the table is acquired.

[0067] That is, the sound processing device according to the seventeenth aspect is a sound processing device in which part or all of the contents of the table are updated in accordance with information acquired while the sound processing device is operating.

[0068] According to such a sound processing device, when information for updating a table is acquired, the table with at least a part updated can be used in subsequent processing.

[0069] In addition, the sound processing device according to the eighteenth aspect is the sound processing device according to the second aspect, in which the reduction processing unit accumulates the amount of calculation in descending order of priority included in the table, and determines as the target sound the sound at or below the rank at which the amount of calculation exceeds a predetermined amount of calculation.

[0070] That is, the sound processing device according to the eighteenth aspect is a sound processing device that applies predetermined processing to sounds with low priority under the constraint of keeping the amount of calculation within a predetermined amount.

[0071] According to such a sound processing device, it is possible to reduce sounds that cannot be processed within a predetermined amount of calculation.

[0072] Furthermore, a sound processing device according to a 19th aspect is a sound processing device according to any one of the 1st to 18th aspects, in which, when there are multiple users or multiple sound source objects, the table reference unit references one table from the multiple tables that corresponds to a combination of the user's position and the sound source object's position.

[0073] That is, the sound processing device according to the nineteenth aspect is a sound processing device that, when there are a plurality of objects or a plurality of listeners, uses a table according to a combination of the object and the listener.

[0074] With such a sound processing device, when there are multiple users or multiple sound source objects, it is possible to refer to one table from multiple tables that corresponds to the combination of the user's position and the sound source object's position.

[0075] Furthermore, the acoustic processing method according to the twentieth aspect is an acoustic processing method executed by a computer, and includes the steps of: acquiring sound information including an acoustic signal and information on the position of a sound source object within a three-dimensional sound field; referring to a table that associates at least one of the user's position and the sound source object's position within the three-dimensional sound field with information for determining a target sound; determining the target sound using the information for determining the target sound obtained by referring to the table; and generating an output sound signal that does not include the signal of the determined target sound by reducing the signal of the determined target sound from multiple sound signals generated when generating an output sound signal from the acoustic signal included in the acquired sound information.

[0076] This can provide the same effects as the sound processing device described above.

[0077] A program according to a twenty-first aspect is a program for causing a computer to execute the above-described acoustic processing method.

[0078] This makes it possible to achieve the same effects as the above-described acoustic processing method using a computer.

[0079] Furthermore, these comprehensive or specific aspects may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0080] Hereinafter, embodiments will be described in detail with reference to the drawings. Note that the embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not recited in independent claims will be described as optional components. Note that each figure is a schematic diagram and is not necessarily an exact illustration. Furthermore, in each figure, substantially identical components are assigned the same reference numerals, and duplicated descriptions may be omitted or simplified.

[0081] In the following description, elements may be assigned ordinal numbers such as first, second, and third. These ordinal numbers are assigned to elements in order to identify them and do not necessarily correspond to a meaningful order. These ordinal numbers may be rearranged, newly added, or removed as appropriate.

[0082] In the following description, an acoustic signal included in sound information may be described, but the acoustic signal may also be referred to as a voice signal or a sound signal. In other words, in the present disclosure, the acoustic signal has the same meaning as the voice signal or the sound signal.

[0083] (Embodiment) [Overview] First, an overview of an audio reproduction system according to an embodiment will be described. Fig. 1 is a schematic diagram showing an example of use of an audio reproduction system according to an embodiment. Fig. 1 shows a user 99 using an audio reproduction system 100.

[0084] The sound reproduction system 100 shown in FIG. 1 is used simultaneously with, for example, a three-dimensional video reproduction device 300. By simultaneously viewing three-dimensional images and three-dimensional sound, the image enhances the auditory sense of realism, and the sound enhances the visual sense of realism, allowing the user to experience the image and sound as if they were actually at the scene where they were captured. For example, when an image (moving image) of people having a conversation is displayed, it is known that even if the localization of the sound image (sound source object) of the conversation sound is not aligned with the person's mouth, the user 99 will perceive the conversation sound as coming from the person's mouth. In this way, the visual information can correct the position of the sound image, and the image and sound can be combined to enhance the sense of realism.

[0085] The three-dimensional video reproduction device 300 is an image display device worn on the head of the user 99. Therefore, the three-dimensional video reproduction device 300 moves integrally with the head of the user 99. For example, as shown in the figure, the three-dimensional video reproduction device 300 is a glasses-type device that is supported by the ears and nose of the user 99.

[0086] The three-dimensional video reproduction device 300 changes the displayed image in accordance with the movement of the user 99's head, thereby making the user 99 perceive the movement of his or her head in the three-dimensional image space. In other words, when an object in the three-dimensional image space is located in front of the user 99, if the user 99 turns to the right, the object moves to the left of the user 99, and if the user 99 turns to the left, the object moves to the right of the user 99. In this way, the three-dimensional video reproduction device 300 moves the three-dimensional image space in the opposite direction to the movement of the user 99.

[0087] The 3D video playback device 300 displays two images with a parallax difference to each of the user's 99's left and right eyes. The user 99 can perceive the three-dimensional position of an object on the image based on the parallax difference between the displayed images. Note that when the user 99 uses the audio playback system 100 with their eyes closed, for example, when using it to play healing sounds for sleep induction, the 3D video playback device 300 does not need to be used at the same time. In other words, the 3D video playback device 300 is not an essential component of the present disclosure. In addition to dedicated video display devices, the 3D video playback device 300 may also be a general-purpose mobile terminal owned by the user 99, such as a smartphone or tablet device.

[0088] Such general-purpose mobile terminals are equipped with not only a display for displaying images but also various sensors for detecting the terminal's posture and movement. Furthermore, they are also equipped with a processor for information processing, and are capable of connecting to a network to transmit and receive information to and from a server device such as a cloud server. In other words, the 3D video playback device 300 and the audio playback system 100 can be realized by combining a smartphone with general-purpose headphones or the like that do not have an information processing function.

[0089] As in this example, the head movement detection function, the video presentation function, the video information processing function for presentation, the sound presentation function, and the sound information processing function for presentation may be appropriately arranged in one or more devices to realize the 3D video reproduction device 300 and the sound reproduction system 100. If the 3D video reproduction device 300 is not required, it is sufficient to appropriately arrange the head movement detection function, the sound presentation function, and the sound information processing function for presentation in one or more devices. For example, the sound reproduction system 100 can be realized by a processing device such as a computer or smartphone having a sound information processing function for presentation, and headphones or the like having a head movement detection function and a sound presentation function.

[0090] The sound reproduction system 100 is a sound presentation device that is worn on the head of the user 99. Therefore, the sound reproduction system 100 moves integrally with the head of the user 99. For example, the sound reproduction system 100 in this embodiment is a so-called over-ear headphone type device. Note that there are no particular limitations on the form of the sound reproduction system 100, and it may be, for example, two earplug-type devices that are worn independently on the left and right ears of the user 99.

[0091] The sound reproduction system 100 changes the sound presented in accordance with the movement of the head of the user 99, thereby making the user 99 perceive as if he or she is moving his or her head within the three-dimensional sound field. For this reason, as described above, the sound reproduction system 100 moves the three-dimensional sound field in the opposite direction to the movement of the user 99.

[0092] Here, when the user 99 moves within the three-dimensional sound field, the position of the sound source object changes relative to the user 99's position within the three-dimensional sound field. As a result, each time the user 99 moves, it is necessary to perform calculation processing based on the positions of the sound source object and the user 99 to generate an output sound signal for playback. Since such processing typically requires a huge amount of processing, in the present disclosure, from the perspective of reducing the amount of processing, an output sound signal is generated and output by reducing the multiple sound signals that constitute the output sound signal used for head-related transfer function convolution. As a result, the number of sound signals used for head-related transfer function convolution is reduced, and a significant reduction in processing volume is expected. In this case, if sound signals to be reduced are determined by performing additional calculations, the processing volume increases by the amount of processing for the additional calculations. Therefore, in the present disclosure, sound signals to be reduced can be determined using as simple a calculation as possible. Therefore, in the present disclosure, sound signals to be reduced can be determined using only simple calculations by referencing a pre-calculated table. In this way, with the present disclosure, it is possible to appropriately determine the sound to be reduced with only simple additional calculations, and to make the effect of reducing the amount of processing obtained by reducing the signal of that sound more pronounced.

[0093] [Configuration] Next, the configuration of the sound reproduction system 100 according to this embodiment will be described with reference to Fig. 2. Fig. 2 is a block diagram showing the functional configuration of the sound reproduction system according to this embodiment.

[0094] As shown in FIG. 2, the sound reproduction system 100 according to this embodiment includes an information processing device 101, a communication module 102, a detector 103, a driver 104, and a database 105.

[0095] The information processing device 101 is an example of an audio processing device, and is an arithmetic device for performing various signal processing in the audio reproduction system 100. The information processing device 101 includes a processor and a memory, such as a computer, and is realized by the processor executing a program stored in the memory. Execution of this program provides functions related to each functional unit described below.

[0096] The information processing device 101 includes an acquisition unit 111, a path calculation unit 121, an output sound generation unit 131, and a signal output unit 141. Details of each functional unit included in the information processing device 101 will be described below together with details of the configuration other than the information processing device 101.

[0097] The communication module 102 is an interface device for accepting input of sound information to the sound reproduction system 100. The communication module 102 includes, for example, an antenna and a signal converter, and receives sound information from an external device via wireless communication. More specifically, the communication module 102 receives, using the antenna, a wireless signal indicating sound information converted into a format for wireless communication, and reconverts the wireless signal into sound information using the signal converter. In this way, the sound reproduction system 100 acquires sound information from the external device via wireless communication. The sound information acquired by the communication module 102 is acquired by the acquisition unit 111. In this way, the acquisition unit 111 is an example of a sound acquisition unit. The sound information is input to the information processing device 101 in the above manner. Note that communication between the sound reproduction system 100 and the external device may be performed via wired communication.

[0098] The sound information acquired by the sound reproduction system 100 is encoded in a predetermined format, such as MPEG-H 3D Audio (ISO / IEC 23008-3). As an example, the encoded sound information includes information about the sound reproduced by the sound reproduction system 100 and information about the localization position when the sound image of the sound is localized at a predetermined position in a three-dimensional sound field (i.e., perceived as sound coming from a predetermined direction). The sound information can also be interpreted as information about a sound source object. In other words, the sound information includes the position of the sound source object in the three-dimensional sound field and the sound produced by the sound source object.

[0099] As described above, sound information is obtained as input data, and includes an audio signal (acoustic signal), which is information about the reproduced sound, and other information, such as information about the position of a sound source object in a three-dimensional sound field. The other information may also include information for defining a three-dimensional sound field. Therefore, the other information may be collectively referred to as information about space (spatial information), including information about the position of a sound source object and information for defining a three-dimensional sound field. When the audio signal is viewed as the main focus, the input data can be said to be sound information in which other information (metadata) is attached to the audio signal. When the spatial information is viewed as the main focus, the input data can be said to be information in which the audio signal is attached to the spatial information. Alternatively, since the input data has both aspects, the input data can also be considered as sound spatial information.

[0100] As a specific example, the sound information includes information about a plurality of sounds including a first reproduced sound and a second reproduced sound, and the sound images of the respective sounds when reproduced are localized so that they are perceived as coming from different positions in a three-dimensional sound field. Therefore, the sound source object of the first reproduced sound is localized at a first position in the three-dimensional sound field, and the sound source object of the second reproduced sound is localized at a second position in the three-dimensional sound field. Thus, the sound information may include a plurality of sounds. In other words, the sound information may include a plurality of audio signals corresponding to the first reproduced sound and the second reproduced sound, respectively, and the positions of a plurality of sound source objects at first and second positions that correspond one-to-one to the plurality of audio signals.

[0101] 3 is a diagram illustrating an example of an audio signal according to an embodiment. For example, as shown in (a) of FIG. 3, the sound information may include an audio signal of a first direct sound arriving from a first position (from a first direction) to the position of the user 99 and an audio signal of a second direct sound arriving from a second position (from a second direction) to the position of the user 99. The sound information immediately after acquisition may include only information about the reproduced sound. In this case, information about a predetermined position may be acquired separately, and the subsequent processing may be performed once the information is collected. As described above, the sound information includes first sound information about the first reproduced sound and second sound information about the second reproduced sound. However, multiple pieces of sound information each including these may be acquired separately and played back simultaneously (i.e., treated as a single piece of sound information), thereby localizing sound images at different positions in a three-dimensional sound field and causing the reproduced sounds to arrive from different directions.

[0102] Alternatively, the sound information may include multiple audio signals and the position of a single sound source object that corresponds to the multiple audio signals in a many-to-one relationship. For example, such sound information is used in a situation where multiple sounds are reproduced from a sound source object. For example, each of the multiple audio signals corresponds to a direct sound that arrives directly from the position of the sound source object to the position of the user 99, and a secondary sound (sound generated by indirect propagation) that accompanies the direct sound and arrives via a path different from that of the direct sound.

[0103] For example, as shown in (b) of FIG. 3 , the sound information immediately after acquisition includes an audio signal related to the direct sound, and is converted into sound information including audio signals such as reverberation, primary reflection, and diffraction through a conversion process for calculating secondary sounds. This conversion process for calculating secondary sounds uses information on the spatial environment conditions of the three-dimensional sound field (e.g., the position, reflection, and diffraction characteristics of objects in the three-dimensional sound field). Thus, secondary sounds are computationally generated from sound information related to a single playback sound based on the spatial environment conditions of the three-dimensional sound field. Therefore, the secondary sounds are not included in the sound information immediately after acquisition, and sound information including these secondary sounds is generated through the conversion process for calculating the secondary sounds. Another secondary sound may be generated from one secondary sound through its propagation. Note that the information on the spatial environment conditions is part of the spatial information and may be acquired together with the audio signal from the input sound information. Alternatively, the audio signal and the spatial information may be acquired separately. That is, the sound information may be acquired from a single file or bitstream, or may be acquired separately by dividing it into multiple files or bitstreams. For example, the audio signal and the spatial information may be acquired from separate files or bitstreams, or the audio signal and the spatial information may each be acquired from a plurality of files or bitstreams.

[0104] The example in Figure 3(b) illustrates that a secondary reflected sound is generated from a primary reflected sound. As shown in the figure, these secondary sounds are assigned tags that identify their relationships with each other, such as parent, child, and grandchild, as information regarding the genealogy of their generation relationships from the direct sound (in other words, the generation system). Alternatively, when the direct sound is defined as the 0th generation, the secondary sounds may be quantified as the number of generations, such as the first generation to which the first reflected sound belongs and the second generation to which the second reflected sound belongs. Note that the number of generations that are allowed to be generated from the direct sound may be set according to the scale of the computational resources.

[0105] As described above, there is no particular limitation on the form of the input sound information, and the sound reproduction system 100 may be provided with an acquisition unit 111 that can accommodate various forms of sound information.

[0106] An example of the acquisition unit 111 will now be described with reference to Fig. 4. Fig. 4 is a block diagram showing the functional configuration of the acquisition unit according to the embodiment. As shown in Fig. 4, the acquisition unit 111 according to the present embodiment includes, for example, an encoded sound information input unit 112, a decoding processing unit 113, a sensing information input unit 114, a table setting unit 115, and a table reference unit 116.

[0107] The encoded sound information input unit 112 is a processing unit to which the coded (in other words, encoded) sound information acquired by the acquisition unit 111 is input. The encoded sound information input unit 112 outputs the input sound information to the decoding processing unit 113. The decoding processing unit 113 is a processing unit that decodes (in other words, decodes) the sound information output from the encoded sound information input unit 112 to generate a playback sound and the position of a sound source object included in the sound information in a format that is used for subsequent processing. The sensing information input unit 114 will be described below together with the function of the detector 103.

[0108] The detector 103 is a device for detecting the speed of movement of the head of the user 99. The detector 103 is configured by combining various sensors used for detecting movement, such as a gyro sensor and an acceleration sensor. In this embodiment, the detector 103 is built into the sound reproduction system 100. However, the detector 103 may be built into an external device, such as a three-dimensional image reproduction device 300 that operates in response to the movement of the head of the user 99 in the same way as the sound reproduction system 100. In this case, the detector 103 does not need to be included in the sound reproduction system 100. Alternatively, the detector 103 may be an external imaging device or the like that captures the movement of the head of the user 99 and detects the movement of the user 99 by processing the captured image.

[0109] The detector 103 is, for example, fixed integrally to the housing of the sound reproduction system 100 and detects the speed of movement of the housing. After the sound reproduction system 100 including the housing is worn by the user 99, it moves integrally with the head of the user 99, and as a result, the detector 103 can detect the speed of movement of the head of the user 99.

[0110] The detector 103 may detect, for example, the amount of head movement of the user 99 as the amount of rotation about at least one of three axes that are orthogonal to each other in three-dimensional space as the rotation axis, or may detect the amount of displacement about at least one of the three axes as the displacement direction. Furthermore, the detector 103 may detect both the amount of rotation and the amount of displacement as the amount of head movement of the user 99.

[0111] The sensing information input unit 114 acquires the movement speed of the user 99's head from the detector 103. More specifically, the sensing information input unit 114 acquires the amount of head movement of the user 99 detected by the detector 103 per unit time as the movement speed. In this way, the sensing information input unit 114 acquires at least one of the rotation speed and the displacement speed from the detector 103. The amount of head movement of the user 99 acquired here is used to determine the position and posture (in other words, coordinates and orientation) of the user 99 in the three-dimensional sound field. Therefore, the acquisition unit 111 also functions as a position acquisition unit via the sensing information input unit 114. In the sound reproduction system 100, the relative position of the sound image object with respect to the user 99 is determined based on the determined coordinates and orientation of the user 99, and sound is reproduced. Specifically, the above functions are realized by the path calculation unit 121 and the output sound generation unit 131.

[0112] The path calculation unit 121 includes an arrival direction calculation function that calculates the relative arrival direction of the reproduced sound from the position of the sound source object to the position of the user 99 based on the determined coordinates and orientation of the user 99, and the conversion process that calculates the secondary sound described above. Therefore, the path calculation unit 121 includes a function that calculates a propagation path from the sound source object and calculates the secondary sound and the arrival direction of the secondary sound that arrives at the position of the user 99 through indirect propagation of the reproduced sound according to the calculated propagation path of the reproduced sound. The arrival direction of the secondary sound includes additional information, such as what object the sound is reflected from in the case of a reflected sound and the attenuation rate at the time of reflection. The additional information is included in the arrival direction of the secondary sound calculated from the input sound information. In other words, the additional information is computationally generated and acquired from the sound information.

[0113] To summarize the spatial information, the spatial information includes further information such as the spatial position of a sound source object in a space (three-dimensional sound field) (information on the position of the sound source object), sound reflection and diffraction characteristics at the sound source object (together with information on the conditions of the spatial environment), and the size of the three-dimensional sound field. Based on the spatial information, the path calculation unit 121 generates secondary sounds depending on which sound source object the reproduced sound is reflected or diffracted by, and calculates, as additional information, the direction from which the secondary sounds arrive and the volume of the secondary sounds after attenuation by reflection or diffraction. The sound information (input data) includes spatial information in the form of audio signals and accompanying metadata. As described above, the spatial information includes, as information other than the audio signal, information necessary to turn the sound into stereophonic sound and position the sound source object in a three-dimensional sound field, and / or information used to calculate information necessary to turn the sound into stereophonic sound and position the sound source object in a three-dimensional sound field.

[0114] The path calculation unit 121 may be realized by any processing as long as it can calculate the arrival direction of the reproduced sound when the reproduced sound reaches the user as a direct sound and can calculate the arrival direction of a secondary sound that arrives at the position of the user 99 due to secondary propagation of the reproduced sound. The path calculation unit 121 determines from which direction in the three-dimensional sound field the reproduced sound and the secondary sound are to be perceived by the user 99 as coming from, based on the coordinates and orientation of the user 99, and processes the sound information so that the output sound signal is perceived as such a sound when it is reproduced.

[0115] The output sound generating unit 131 is a processing unit that processes information about the reproduced sound included in the sound information to generate an output sound signal.

[0116] An example of the output sound generation unit 131 will now be described with reference to Fig. 5. Fig. 5 is a block diagram showing the functional configuration of the output sound generation unit according to an embodiment. As shown in Fig. 5, the output sound generation unit 131 in this embodiment includes, for example, a reduction processing unit 132, which includes a culling unit 133 and an integration unit 134.

[0117] The reduction processing unit 132 is a processing unit that processes sound information by the path calculation unit 121, etc., and determines from multiple sound signals, such as direct sound, reverberant sound, (primary, secondary and higher-order) reflected sound, and indirect sound such as diffracted sound, sound signals that are unlikely to cause an auditory difference even if reduced, i.e., sound signals that are unlikely to cause the user 99 to perceive a deterioration in sound, and reduces those signals.

[0118] The reduction processing unit 132 uses the culling unit 133 to stop the generation of a sound signal, or to perform a culling process that discards the generated sound signal, so that the sound signal is not included in a subsequent output sound signal. The culling unit 133 is a processing unit that discards the specific sound signal determined as the sound to be reduced in this way. Note that discarding here is in a broad sense, including discarding a signal by stopping the generation itself.

[0119] Furthermore, the reduction processing unit 132 discards two or more sound signals using the integrating unit 134, and instead integrates the discarded sound signals into a smaller number of virtual sounds to generate one or more virtual sounds that virtually replace the two or more sounds, so that the two or more sound signals are not included in the subsequent output sound signal, and fewer virtual sound signals are included in the subsequent output sound signal. The integrating unit 134 is a processing unit that discards two or more specific sound signals determined as sounds to be reduced in this way, and generates a smaller number of virtual sound signals to replace them.

[0120] Here, the reduction processing unit 132 determines a specific sound to be reduced based on the table prepared and referenced by the table setting unit 115 and the table reference unit 116 shown in FIG.

[0121] The table setting unit 115 acquires a table from an external device, or sets a table by table generation processing and stores it in a storage unit (not shown). The table set by the table setting unit 115 is a database that associates at least one of the position of the user 99 and the position of the sound source object in the three-dimensional sound field with information for determining a target sound. The table setting unit 115 sets a table received from outside as a table to be used by the reduction processing unit 132, or, when information for updating a table is acquired, updates the already set table using the information so that the table to be used by the reduction processing unit 132 is updated before and after the acquisition.

[0122] The table reference unit 116 is a processing unit that obtains information for determining the target sound by referencing a table. For example, by querying (i.e., referencing) the table using the position of the user 99 and the position of the sound source object as arguments, information for determining the sound to be reduced can be obtained in response. The information for determining the sound to be reduced may be information that directly indicates the target sound, or may be information that indirectly indicates the target sound by being subjected to a calculation process using the information for determining the target sound. The information for determining the target sound will be described in more detail in the examples below.

[0123] Referring again to FIG. 2 , the output sound generation unit 131 acquires a head-related transfer function used to generate an output sound signal from the database 105. The database 105 is an information storage device that functions both as a storage device for storing information and as a storage controller that reads out the stored information and outputs it to an external component. The database 105 stores a head-related transfer function for each direction of arrival of the sound from the user 99. The head-related transfer functions included in the database 105 are a set of general-purpose head-related transfer functions that can be used by everyone, a set of head-related transfer functions optimized for each individual user 99, or a set of head-related transfer functions that are publicly available. The database 105 receives an inquiry from the output sound generation unit 131 using the direction of arrival as a query, and outputs a head-related transfer function corresponding to the direction of arrival to the output sound generation unit 131. In addition, the output sound generation unit 131 may output the entire set of head-related transfer functions, or may output the characteristics of the set of head-related transfer functions itself.

[0124] The signal output unit 141 is a functional unit that outputs the generated output sound signal to the driver 104. The signal output unit 141 generates a waveform signal by performing signal conversion from a digital signal to an analog signal based on the output sound signal, and then causes the driver 104 to generate sound waves based on the waveform signal, thereby presenting sound to the user 99. The driver 104 includes, for example, a diaphragm and a drive mechanism such as a magnet and a voice coil. The driver 104 operates the drive mechanism in response to the waveform signal, causing the drive mechanism to vibrate the diaphragm. In this way, the driver 104 generates sound waves by vibrating the diaphragm in response to the output sound signal (this means "reproducing" the output sound signal; in other words, "reproducing" does not include perception by the user 99). The sound waves propagate through the air and reach the ears of the user 99, and the user 99 perceives the sound.

[0125] [Another Configuration Example] In the above example, the sound reproduction system 100 according to the present embodiment is an audio presentation device, and has been described as including an information processing device 101, a communication module 102, a detector 103, a database 105, and a driver 104. However, the functions of the sound reproduction system 100 may be realized by a plurality of devices or by a single device. Specific examples will be described using Figures 6 to 14. Figures 6 to 14 are diagrams for explaining other examples of the sound reproduction system according to the embodiment.

[0126] For example, the information processing device 601 may be included in the audio presentation device 602, and the audio presentation device 602 may perform both acoustic processing and sound presentation. Alternatively, the information processing device 601 and the audio presentation device 602 may share the acoustic processing described in the present disclosure, or a server connected to the information processing device 601 or the audio presentation device 602 via a network may perform part or all of the acoustic processing described in the present disclosure.

[0127] In the above explanation, the information processing device 601 is referred to as such, but when the information processing device 601 performs acoustic processing by decoding a bitstream generated by encoding at least a portion of the data of an audio signal or spatial information used for acoustic processing, the information processing device 601 may be referred to as a decoding device, and the acoustic reproduction system 100 (i.e., the stereophonic sound reproduction system 600 in the figure) may be referred to as a decoding processing system.

[0128] Here, an example will be described in which the sound reproduction system 100 functions as a decoding processing system.

[0129] <Example of Encoding Device> FIG. 7 is a functional block diagram showing the configuration of an encoding device 700 that is an example of an encoding device according to the present disclosure.

[0130] Input data 701 is data to be coded, including spatial information and / or an audio signal, that is input to an encoder 702. Details of the spatial information will be described later.

[0131] The encoder 702 encodes the input data 701 to generate encoded data 703. The encoded data 703 is, for example, a bit stream generated by the encoding process.

[0132] The memory 704 stores the encoded data 703. The memory 704 may be, for example, a hard disk or a solid-state drive (SSD), or may be any other storage device.

[0133] In the above description, a bitstream generated by an encoding process is given as an example of the encoded data 703 stored in memory 704. However, data other than a bitstream may also be used. For example, the encoding device 700 may convert a bitstream into a predetermined data format and store the converted data in memory 704. The converted data may be, for example, a file or multiplexed stream storing one or more bitstreams. Here, the file may have a file format such as ISOBMFF (ISO Base Media File Format). The encoded data 703 may also be in the form of multiple packets generated by dividing the bitstream or file. When converting the bitstream generated by the encoder 702 into data other than the bitstream, the encoding device 700 may include a conversion unit (not shown), or the conversion process may be performed by a CPU (Central Processing Unit).

[0134] <Example of Decoding Device> FIG. 8 is a functional block diagram showing the configuration of a decoding device 800 that is an example of a decoding device according to the present disclosure.

[0135] The memory 804 stores, for example, the same data as the coded data 703 generated by the coding device 700. The memory 804 reads out the stored data and inputs it as input data 803 to the decoder 802. The input data 803 is, for example, a bitstream to be decoded. The memory 804 may be, for example, a hard disk or an SSD, or may be another storage device.

[0136] Note that the decoding device 800 may not use the data stored in the memory 804 as input data 803 as is, but may convert the read data and generate converted data as input data 803. The data before conversion may be, for example, multiplexed data storing one or more bitstreams. Here, the multiplexed data may be, for example, a file having a file format such as ISOBMFF. The data before conversion may also be in the form of multiple packets generated by dividing the bitstream or file. When converting data different from the bitstream read from the memory 804 into a bitstream, the decoding device 800 may be provided with a conversion unit (not shown), or the conversion process may be performed by a CPU.

[0137] Decoder 802 decodes input data 803 to generate audio signal 801 that is presented to a listener.

[0138] <Another Example of Encoding Device> Fig. 9 is a functional block diagram showing the configuration of an encoding device 900, which is another example of an encoding device of the present disclosure. In Fig. 9, components having the same functions as those in Fig. 7 are assigned the same reference numerals as those in Fig. 7, and descriptions of these components will be omitted.

[0139] The coding device 900 differs from the coding device 700 in that the coding device 700 includes a memory 704 for storing coded data 703, whereas the coding device 900 includes a transmitting unit 901 for transmitting coded data 703 to the outside.

[0140] The transmitter 901 transmits a transmission signal 902 to another device or a server based on the encoded data 703 or data in another data format generated by converting the encoded data 703. The data used to generate the transmission signal 902 is, for example, the bit stream, multiplexed data, file, or packet described in the encoding device 700.

[0141] <Another Example of Decoding Device> Fig. 10 is a functional block diagram showing the configuration of a decoding device 1000, which is another example of a decoding device of the present disclosure. In Fig. 10, components having the same functions as those in Fig. 8 are assigned the same reference numerals as those in Fig. 8, and descriptions of these components will be omitted.

[0142] The decoding device 800 differs from the decoding device 800 in that the decoding device 800 includes a memory 804 for reading out input data 803, whereas the decoding device 1000 includes a receiving unit 1001 for receiving input data 803 from the outside.

[0143] The receiving unit 1001 receives a received signal 1002, acquires received data, and outputs input data 803 to be input to the decoder 802. The received data may be the same as the input data 803 to be input to the decoder 802, or may be data in a data format different from that of the input data 803. If the received data is data in a data format different from that of the input data 803, the receiving unit 1001 may convert the received data into the input data 803, or a conversion unit or CPU (not shown) included in the decoding device 1000 may convert the received data into the input data 803. The received data is, for example, a bit stream, multiplexed data, a file, or a packet, as described in the encoding device 900.

[0144] <Functional Description of Decoder> FIG. 11 is a functional block diagram showing the configuration of a decoder 1100, which is an example of the decoder 802 in FIG. 8 or FIG.

[0145] The input data 803 is an encoded bitstream, and includes encoded audio data, which is an encoded audio signal, and metadata used for acoustic processing.

[0146] The spatial information management unit 1101 acquires metadata included in the input data 803 and analyzes the metadata. The metadata includes information describing elements that act on sounds arranged in a sound space. The spatial information management unit 1101 manages spatial information necessary for acoustic processing obtained by analyzing the metadata and provides the spatial information to the rendering unit 1103. Note that, although the information used for acoustic processing is referred to as spatial information in this disclosure, it may be called by other names. The information used for acoustic processing may be called, for example, sound space information or scene information. Furthermore, when the information used for acoustic processing changes over time, the spatial information input to the rendering unit 1103 may be called a space state, a sound space state, a scene state, or the like.

[0147] Furthermore, spatial information may be managed for each sound space or for each scene. For example, when different rooms are represented as virtual spaces, each room may be managed as a scene of a different sound space, or even if the same space is represented, spatial information may be managed as different scenes depending on the scene being represented. In managing spatial information, an identifier for identifying each piece of spatial information may be assigned. The spatial information data may be included in a bitstream, which is one form of input data 803, or the bitstream may include an identifier for the spatial information, and the spatial information data may be acquired from a source other than the bitstream. When the bitstream includes only the identifier for the spatial information, the identifier for the spatial information may be used during rendering to acquire the spatial information data stored in the memory of the acoustic signal processing device or an external server as input data.

[0148] Note that the information managed by the spatial information management unit 1101 is not limited to information included in the bitstream. For example, the input data 803 may include data indicating the characteristics or structure of a space acquired from a software application or server providing VR or AR, as data not included in the bitstream. Furthermore, for example, the input data 803 may include data indicating the characteristics or position of a listener or object, as data not included in the bitstream. Furthermore, the input data 803 may include, as information indicating the position of the listener, information acquired by a sensor provided in a terminal including a decoding device, or information indicating the position of the terminal estimated based on information acquired by the sensor. In other words, the spatial information management unit 1101 may communicate with an external system or server to acquire spatial information and the position of the listener. Furthermore, the spatial information management unit 1101 may acquire clock synchronization information from an external system and execute a process of synchronizing with the clock of the rendering unit 1103. Note that the space in the above description may be a virtually formed space, i.e., a VR space, or may be a real space or a virtual space corresponding to a real space, i.e., an AR space or an MR (Mixed Reality) space. The virtual space may also be called a sound field or a sound space. Furthermore, the information indicating a position in the above description may be information such as coordinate values ​​indicating a position within a space, information indicating a relative position with respect to a predetermined reference position, or information indicating the movement or acceleration of a position within a space.

[0149] The audio data decoder 1102 decodes the encoded audio data included in the input data 803 to obtain an audio signal.

[0150] The encoded audio data acquired by the stereophonic sound reproduction system 600 is a bitstream encoded in a predetermined format such as MPEG-H 3D Audio (ISO / IEC 23008-3). Note that MPEG-H 3D Audio is merely one example of an encoding method that can be used to generate the encoded audio data included in the bitstream, and the encoded audio data may be included in a bitstream encoded in another encoding method. For example, the encoding method used may be a lossy codec such as MP3 (MPEG-1 Audio Layer-3), AAC (Advanced Audio Coding), WMA (Windows Media Audio), AC3 (Audio Codec-3), or Vorbis, or a lossless codec such as ALAC (Apple Lossless Audio Codec) or FLAC (Free Lossless Audio Codec), or any other encoding method may be used. For example, PCM (Pulse Code Modulation) data may be a type of encoded audio data. In this case, the decoding process may be, for example, a process of converting an N-bit binary number into a number format (e.g., floating-point format) that can be processed by the rendering unit 1103, where the number of quantization bits of the PCM data is N.

[0151] The rendering unit 1103 receives an audio signal and spatial information as input, performs acoustic processing on the audio signal using the spatial information, and outputs an audio signal 801 after acoustic processing.

[0152] Before starting rendering, the spatial information management unit 1101 reads metadata of the input signal, detects rendering items such as objects or sounds defined in the spatial information, and transmits the detected items to the rendering unit 1103. After starting rendering, the spatial information management unit 1101 grasps changes over time in the spatial information and the position of the listener, and updates and manages the spatial information. The spatial information management unit 1101 then transmits the updated spatial information to the rendering unit 1103. The rendering unit 1103 generates and outputs an audio signal to which acoustic processing has been applied based on the audio signal included in the input data and the spatial information received from the spatial information management unit 1101.

[0153] The spatial information update process and the audio signal output process with added acoustic processing may be executed in the same thread, or the spatial information management unit 1101 and the rendering unit 1103 may be allocated to independent threads. When the spatial information update process and the audio signal output process with added acoustic processing are executed in different threads, the thread startup frequency may be set individually, or the processes may be executed in parallel.

[0154] By having the spatial information management unit 1101 and the rendering unit 1103 execute their processes in different, independent threads, computational resources can be preferentially allocated to the rendering unit 1103. Therefore, in the case of sound output processing in which even a slight delay cannot be tolerated, for example, sound output processing in which a delay of even one sample (0.02 msec) would cause a popping noise, can be safely performed. In this case, the allocation of computational resources to the spatial information management unit 1101 is limited. However, compared to audio signal output processing, updating spatial information is a less frequent process (e.g., a process such as updating the listener's facial orientation). Therefore, unlike audio signal output processing, updating spatial information does not necessarily require an instantaneous response, and therefore limiting the allocation of computational resources does not significantly affect the acoustic quality provided to the listener.

[0155] The space information may be updated periodically at preset times or intervals, or when preset conditions are met. The space information may be updated manually by a listener or a sound space manager, or may be triggered by a change in an external system. For example, when a listener operates a controller to instantly warp the position of their avatar, instantly advance or rewind the time, or when a virtual space manager suddenly changes the environment of the space, the thread in which the space information management unit 1101 is located may be started as a one-off interrupt process in addition to being started periodically.

[0156] The role of the information update thread that executes the spatial information update process is, for example, to update the position or orientation of the listener's avatar placed in the virtual space based on the position or orientation of the VR goggles worn by the listener, and to update the position of objects moving in the virtual space. These tasks are handled within a processing thread that runs relatively infrequently, on the order of several tens of Hz. Processing to reflect the properties of direct sound may be performed in such an infrequently occurring processing thread. This is because the properties of direct sound change less frequently than the frequency of audio processing frames for audio output. Doing so can relatively reduce the computational load of the process and also avoid the risk of pulsive noise occurring when information is updated at an unnecessarily fast frequency.

[0157] FIG. 12 is a functional block diagram showing the configuration of a decoder 1200, which is another example of the decoder 802 in FIG. 8 or 10.

[0158] Figure 12 differs from Figure 11 in that the input data 803 includes an unencoded audio signal rather than encoded audio data. The input data 803 includes a bitstream including metadata and an audio signal.

[0159] The spatial information management unit 1201 is the same as the spatial information management unit 1101 in FIG. 11, and therefore a description thereof will be omitted.

[0160] The rendering unit 1202 is the same as the rendering unit 1103 in FIG. 11, and therefore a description thereof will be omitted.

[0161] In the above description, the configuration in Fig. 12 is called a decoder, but it may also be called an audio processing unit that performs audio processing. Furthermore, a device including an audio processing unit may also be called an audio processing device rather than a decoding device. Furthermore, the audio signal processing device (information processing device 601) may also be called an audio processing device.

[0162] <Physical Configuration of Encoding Apparatus> Fig. 13 is a diagram showing an example of the physical configuration of an encoding apparatus. The encoding apparatus shown in Fig. 13 is an example of the encoding apparatuses 700 and 900 described above.

[0163] The encoding device of FIG. 13 includes a processor, a memory, and a communication IF.

[0164] The processor may be, for example, a CPU (Central Processing Unit), a DSP (Digital Signal Processor), or a GPU (Graphics Processing Unit), and the CPU, DSP, or GPU may execute a program stored in a memory to perform the encoding process of the present disclosure. Alternatively, the processor may be a dedicated circuit that performs signal processing on an audio signal, including the encoding process of the present disclosure.

[0165] The memory may be configured, for example, by RAM (Random Access Memory) or ROM (Read Only Memory). The memory may also include a magnetic storage medium such as a hard disk or a semiconductor memory such as an SSD (Solid State Drive). The term "memory" may also refer to an internal memory built into a CPU or GPU.

[0166] The communication IF (Interface) is a communication module compatible with a communication method such as Bluetooth (registered trademark) or WIGIG (registered trademark). The encoding device has a function of communicating with other communication devices via the communication IF and transmits an encoded bitstream.

[0167] The communication module is composed of, for example, a signal processing circuit and an antenna corresponding to the communication method. In the above example, Bluetooth (registered trademark) or WIGIG (registered trademark) is used as the communication method, but other communication methods such as LTE (Long Term Evolution), NR (New Radio), or Wi-Fi (registered trademark) may also be supported. Furthermore, the communication IF may be a wired communication method such as Ethernet (registered trademark), USB (Universal Serial Bus), or HDMI (registered trademark) (High-Definition Multimedia Interface) instead of the wireless communication method described above.

[0168] <Physical configuration of audio signal processing device> Fig. 14 is a diagram showing an example of the physical configuration of an audio signal processing device. Note that the audio signal processing device in Fig. 14 may be a decoding device. Furthermore, part of the configuration described here may be provided in the audio presentation device 602. Furthermore, the audio signal processing device shown in Fig. 14 is an example of the audio signal processing device 601 described above.

[0169] The acoustic signal processing device of FIG. 14 includes a processor, a memory, a communication IF, a sensor, and a speaker.

[0170] The processor may be, for example, a CPU (Central Processing Unit), a DSP (Digital Signal Processor), or a GPU (Graphics Processing Unit), and the CPU, DSP, or GPU may execute a program stored in a memory to perform the audio processing or decoding processing of the present disclosure. Alternatively, the processor may be a dedicated circuit that performs signal processing on audio signals, including the audio processing of the present disclosure.

[0171] The memory may be configured, for example, by RAM (Random Access Memory) or ROM (Read Only Memory). The memory may also include a magnetic storage medium such as a hard disk or a semiconductor memory such as an SSD (Solid State Drive). The term "memory" may also refer to an internal memory built into a CPU or GPU.

[0172] The communication IF (Interface) is a communication module compatible with a communication method such as Bluetooth (registered trademark) or WIGIG (registered trademark). The acoustic signal processing device shown in Fig. 14 has a function of communicating with other communication devices via the communication IF and acquires a bitstream to be decoded. The acquired bitstream is stored in a memory, for example.

[0173] The communication module is composed of, for example, a signal processing circuit and an antenna corresponding to the communication method. In the above example, Bluetooth (registered trademark) or WIGIG (registered trademark) is used as the communication method, but other communication methods such as LTE (Long Term Evolution), NR (New Radio), or Wi-Fi (registered trademark) may also be supported. Furthermore, the communication IF may be a wired communication method such as Ethernet (registered trademark), USB (Universal Serial Bus), or HDMI (registered trademark) (High-Definition Multimedia Interface) instead of the wireless communication method described above.

[0174] The sensor performs sensing to estimate the position or orientation of the listener. Specifically, the sensor estimates the position and / or orientation of the listener based on one or more detection results of the position, orientation, movement, velocity, angular velocity, or acceleration of a part or the entire body of the listener, such as the head, and generates position information indicating the position and / or orientation of the listener. Note that the position information may be information indicating the position and / or orientation of the listener in real space, or information indicating a displacement of the position and / or orientation of the listener based on the position and / or orientation of the listener at a predetermined time. Furthermore, the position information may be information indicating the position and / or orientation relative to the stereophonic sound reproduction system or an external device equipped with the sensor.

[0175] The sensor may be, for example, an imaging device such as a camera or a ranging device such as LiDAR (Light Detection and Ranging), and may capture an image of the listener's head movement and detect the movement of the listener's head by processing the captured image. Alternatively, the sensor may be a device that performs position estimation using wireless signals of any frequency band, such as millimeter waves.

[0176] The audio signal processing device shown in Fig. 14 may acquire position information from an external device equipped with a sensor via a communication IF. In this case, the audio signal processing device does not need to include a sensor. Here, the external device is, for example, the audio presentation device 602 described in Fig. 6 or a 3D video playback device worn on the listener's head. In this case, the sensor is configured by combining various sensors such as a gyro sensor and an acceleration sensor.

[0177] The sensor may, for example, detect the angular velocity of rotation around at least one of three mutually perpendicular axes in the sound space as the axis of rotation as the speed of movement of the listener's head, or may detect the acceleration of displacement with at least one of the three axes as the direction of displacement.

[0178] For example, the sensor may detect the amount of rotation about at least one of three mutually orthogonal axes in the sound space as the rotation axis, or the amount of displacement about at least one of the three axes as the displacement direction, as the amount of movement of the listener's head. Specifically, the sensor detects 6 DoF (position (x, y, z) and angle (yaw, pitch, roll)) as the position of the listener. The sensor is configured by combining various sensors used for detecting movement, such as a gyro sensor and an acceleration sensor.

[0179] The sensor may be any device capable of detecting the position of the listener, such as a camera or a GPS (Global Positioning System) receiver. Alternatively, the sensor may use location information obtained by performing self-position estimation using LiDAR (Laser Imaging Detection and Ranging). For example, when the audio signal reproduction system is implemented by a smartphone, the sensor is built into the smartphone.

[0180] The sensor may also include a temperature sensor such as a thermocouple that detects the temperature of the acoustic signal processing device shown in Figure 14, and a sensor that detects the remaining charge of a battery provided in or connected to the acoustic signal processing device.

[0181] A speaker has, for example, a diaphragm, a drive mechanism such as a magnet or a voice coil, and an amplifier, and presents an audio signal after acoustic processing to a listener as sound. The speaker operates the drive mechanism in response to the audio signal (more specifically, a waveform signal indicating the waveform of the sound) amplified by the amplifier, and the drive mechanism vibrates the diaphragm. In this way, the diaphragm vibrating in response to the audio signal generates sound waves, which propagate through the air and reach the listener's ears, causing the listener to perceive the sound.

[0182] Note that, although the description has been given here of an example in which the acoustic signal processing device shown in FIG. 14 includes a speaker and presents an audio signal after acoustic processing via the speaker, the means for presenting the audio signal is not limited to the above configuration. For example, the audio signal after acoustic processing may be output to an external audio presentation device 602 connected via a communication module. Communication via the communication module may be wired or wireless. As another example, the acoustic signal processing device shown in FIG. 14 may include a terminal for outputting an analog audio signal, and a cable such as earphones may be connected to the terminal to present the audio signal from the earphones. In the above case, the audio signal is reproduced by headphones, earphones, a head-mounted display, a neck speaker, a wearable speaker, a surround speaker composed of multiple fixed speakers, or the like, which is worn on the head or part of the body of the listener, which is the audio presentation device 602.

[0183] <Functional Description of Rendering Unit, Example 1> Examples 1 to 3 will be used below to describe examples of detailed configurations of the rendering units 1103 and 1202 in Fig. 11 and Fig. 12. Figs. 15 to 34 are diagrams for explaining specific examples of the sound reproduction system according to Example 1 of the embodiment.

[0184] In the first embodiment, when the shape of a room, which is a three-dimensional sound field, the materials (reflectivity, absorption, etc.) of the walls used in the room, and the shapes and materials (same as above) of obstacles in the room are given, the priorities (also called priority levels) of multiple sounds such as direct sound, reflected sound, reverberant sound, and diffracted sound are determined in advance for the possible positional relationship between the listener (user 99) and the sound source object, and a culling table is created in advance as a table for culling processing. In other words, the characteristic is that the sounds to be reduced by the culling processing are determined in the order of priority shown in the table. Note that, although the individual descriptions are based on the assumption that culling processing is used as the reduction processing, integration processing may also be used as the reduction processing.

[0185] Next, we will explain Fig. 15. Fig. 15 is a block diagram of a decoder (i.e., a rendering unit 1520 and a spatial information management unit 1510) according to this example. The basic idea of ​​this example is to refer to a culling table based on the position of the sound source object and the position of the listener, identify the priority of sounds that will reach the listener at those positions, and generate sounds that will reach the listener according to that priority. This makes it possible to cancel in advance the generation process of sounds that have low importance (priority) among the sounds that will reach the listener, thereby reducing the amount of calculation.

[0186] First, metadata is provided to the decoder. The metadata configuration is shown in Fig. 16. The spatial information mainly represents information about the space in which immersive audio is provided to the listener, such as the shape of the room, the characteristics of the wall materials (such as sound reflection and absorption rates), the characteristics of the materials of obstacles (such as sound reflection and absorption rates), and information about their placement.

[0187] The object information mainly represents information about the position and orientation of the sound source object and the sound emitted from the sound source object.

[0188] The listener information mainly represents information relating to the position and orientation of the listener.

[0189] The culling table is referenced based on the object position and listener position contained in the metadata to obtain the priority of the sounds that will reach the listener. Sounds that will reach the listener are generated according to this priority.

[0190] Based on the priority, control information for controlling whether or not to operate each of the direct sound generation unit 1521, the reverberation sound generation unit 1522, the reflected sound generation unit 1523, and the diffracted sound generation unit 1524 is determined, and the control information is output to the switch unit 1513.

[0191] The metadata is provided to each of the direct sound generation unit 1521, reverberation sound generation unit 1522, reflected sound generation unit 1523, and diffracted sound generation unit 1524 via the switch unit 1513. According to the priority obtained by the culling table reference unit 1511 with reference to the culling table, the control information determination unit 1512 determines the control information for each generation unit (that is, the direct sound generation unit 1521, reverberation sound generation unit 1522, reflected sound generation unit 1523, and diffracted sound generation unit 1524; the same applies hereinafter when simply referred to as a "sound generation unit" or a "generation unit").

[0192] Audio signals are also provided to direct sound generation unit 1521, reverberation sound generation unit 1522, reflected sound generation unit 1523, and diffracted sound generation unit 1524. The audio signals may be generated by decoding encoded audio data included in the input data using an audio data decoder (not shown), and provided to direct sound generation unit 1521, reverberation sound generation unit 1522, reflected sound generation unit 1523, and diffracted sound generation unit 1524, respectively, or audio data included in the input data may be provided to direct sound generation unit 1521, reverberation sound generation unit 1522, reflected sound generation unit 1523, and diffracted sound generation unit 1524, respectively.

[0193] Next, the switch unit 1513 will be described with reference to Fig. 17, which is a block diagram of the switch unit.

[0194] When metadata is input and a switch is turned on, the metadata is given to the generator that will be switched on. Control information for whether the switch is on or off is determined based on priority information as described above. This determination method can be, for example, to turn on the switches for a predetermined number of generators ranked from the top of the priority obtained from the culling table, and turn off the switches for the other generators.

[0195] The generators that are switched on receive the metadata, generate direct sound, reverberant sound, reflected sound, or diffracted sound, and output the generated sound to the sound generator 1525 .

[0196] The sound generator 1525 performs sound processing such as HRTF (Head Related Transfer Function) on the signals output from each generator, and outputs the resulting signals as output signals. This sound processing is adapted to the output format to the listener, such as headphones or multi-channel speakers, and provides the output signals to the listener.

[0197] Although the decoder described above has the generators configured in parallel, the configuration is not limited to this, and the generators may also be configured in series. The serial configuration will be described later.

[0198] Next, a flowchart of the operation of the decoder in the above configuration is shown in FIG.

[0199] First, it is determined whether metadata has been input (S1801). If metadata has been input (Yes in S1801), the culling table is referenced (S1802). If metadata has not been input (No in S1801), the process ends.

[0200] The culling table is referenced based on the positions of the listener and sound source object (S1802) to obtain the priority of each generation unit.

[0201] Next, control information specifying whether the switch of each generation unit is on or off is determined based on the priority information (S1803). For example, the determination method may be to turn on the switches of a predetermined number of generation units in the order of priority obtained from the culling table, and to turn off the switches of the other generation units.

[0202] Next, the on / off status of the switch of the direct sound generation unit 1521 is checked (S1804), and if the switch is on (Yes in S1804), direct sound is generated (S1805), and if the switch is off (No in S1804), direct sound is not generated (S1805 is skipped).

[0203] Similarly, the switch of the reverberation sound generation unit 1522 is referenced (S1806), and if the switch is on (Yes in S1806), a reverberation sound is generated (S1807); if the switch is off (No in S1806), no reverberation sound is generated (S1807 is skipped). The switch of the reflected sound generation unit 1523 is referenced (S1808), and if the switch is on (Yes in S1808), a reflected sound is generated (S1809); if the switch is off (No in S1808), no reflected sound is generated (S1809 is skipped). The switch of the diffracted sound generation unit 1524 is referenced (S1810), and if the switch is on (Yes in S1810), a diffracted sound is generated (S1811); if the switch is off (No in S1810), no diffracted sound is generated (S1811 is skipped).

[0204] Finally, the generated signals are subjected to stereophonic signal processing such as convolution processing of head-related transfer functions to generate stereophonic signals (S1812), which are output to a device used by the listener, such as headphones.

[0205] Then, the process returns to step S1801, where it is determined whether new metadata is to be input.

[0206] The following will be described with reference to Fig. 19. Fig. 19 is a diagram showing the concept of a culling table.

[0207] 19 shows a state in which the space in which the sound source object 98 and the listener (user 99) exist is divided into three-dimensional grids. In this case, assuming that the sound source object and the listener are located on any grid point, the culling table stores the priority of the sounds that reach the listener for the relationship between those grid points. In other words, here, the positions of the sound source object and the listener are converted into grid point positions and the culling table is referenced.

[0208] 19 is shown on the XY plane for convenience, but the priority of the sounds (direct sound (a), reverberant sounds (c) to (g), reflected sound (b), and diffracted sound (h)) that reach the listener when the position of the sound source object 98 is (X, Y) = (2, 2) and the position of the listener is (X, Y) = (2, 7) is obtained by referring to the culling table. Note that an obstacle 97 is shown in the figure as one of the factors that cause secondary sounds to be generated.

[0209] If at least one of the sound source object and the listener is not located on a three-dimensional grid point, the culling table is referenced, assuming that at least one of the sound source object and the listener is located at the grid point closest to at least one of the sound source object and the listener (and converting the grid point position in this way).

[0210] Next, we will explain Fig. 20. Fig. 20 is a diagram showing the configuration of a culling table. In Fig. 20, the left column shows the coordinate position (X, Y, Z) of the sound source object, the center column shows the coordinate position (X, Y, Z) of the listener, and the right column shows the priority corresponding to the combination of the coordinate position of the sound source object and the coordinate position of the listener. The priority is as follows: A indicates direct sound, B indicates reverberant sound, C indicates reflected sound, and D indicates diffracted sound, with the priority increasing further to the left and decreasing further to the right (descending order from left to right).

[0211] By referring to the culling table, it is possible to obtain the priority based on the position of the sound source object and the position of the listener. For example, in FIG. 20, if the position of the sound source object is (O ｘ（ｌ） ,O ｙ（ｍ） ,O ｚ（ｎ） ), the listener position is (L ｘ（ｌ） ,L ｙ（ｍ） ,L ｚ（ｎ） ), the priority order (A, B, D, C) is obtained. In other words, the priority order is obtained as follows: direct sound > reverberant sound > diffracted sound > reflected sound. Note that in the figure, L in the subscript brackets is the coordinate number on the X axis, M in the subscript brackets is the coordinate number on the Y axis, and N in the subscript brackets is the coordinate number on the Z axis.

[0212] A culling table can be created in advance. A specific method for this is to calculate the direct sound, reverberation sound, reflection sound, and diffracted sound that actually reach the listener for all combinations of the position of the sound source object and the position of the listener, and then compare the importance of these signals with indices that take into account auditory characteristics, such as the magnitude of energy or energy corrected for auditory characteristics, to determine priority. This requires a huge amount of calculation, but because the culling table can be created in advance, it is possible to realize this without having to consider hardware processing capacity limitations or real-time performance during rendering.

[0213] Alternatively, a content creator may design and create a culling table, partly or entirely.

[0214] The culling table may be stored in advance in the decoder as the initial state of the decoder. In this case, it is possible to reduce the amount of calculation using the culling table without any special processing.

[0215] Alternatively, the culling tables may be obtained during the decoder initialization process, or some or all of the culling tables may be updated while the decoder is running, as will be discussed below.

[0216] Note that while we have described a culling table designed based on both the position of the sound source object and the position of the listener, in order to reduce the amount of calculation required when designing the culling table and the memory required for the culling table, the culling table may also be designed based on either the position of the sound source object or the position of the listener.

[0217] Next, Fig. 21 will be described. Fig. 21 is a diagram showing a culling table according to another embodiment. Fig. 21 shows a diagram similar to Fig. 20, and differs in that in this example, the culling table is constructed by dividing reflected sounds into primary reflected sounds C1 that hit a wall or obstacle once and reach the listener, secondary reflected sounds C2 that hit a wall or obstacle twice and reach the listener, and higher-order reflected sounds C3 that hit a wall or obstacle three or more times and reach the listener.

[0218] Excluding direct sound, reflected sound has more energy than reverberation and diffraction sound, which are sounds that reach other listeners. Therefore, more appropriate culling can be performed by constructing a culling table that takes into account sounds of secondary reflections and above.

[0219] 21, the priority of each sound that reaches the listener is acquired by referring to the position of the sound source object and the position of the listener. The sounds that reach the listener are direct sound, reverberant sound, first reflected sound, second reflected sound, higher-order reflected sound, and diffracted sound, and are the sounds that are the subject of priority acquisition.

[0220] For example, in FIG. 21, if the position of the sound source object is (O ｘ（ｌ） ,O ｙ（ｍ） ,O ｚ（ｎ） ), the listener position is (L ｘ（Ｌ-１） ,L ｙ（Ｍ-１） ,L ｚ（Ｎ-１） ), the priority order (B, C1, A, C2, D, C3) is obtained. That is, the priority order is obtained as follows: reverberant sound > primary reflected sound > direct sound > secondary reflected sound > diffracted sound > higher-order reflected sound.

[0221] 22 and 23 will be described. In the above description, as shown in Fig. 19, the space in which the sound source objects and listeners exist is divided into equally spaced grids in three dimensions (i.e., into unit spaces of a fixed size), and the culling table is defined on the assumption that the sound source objects and listeners are located on arbitrary grid points.

[0222] In the example of Figure 22, the three-dimensional space is divided into voxels of arbitrary size (i.e., unit spaces of multiple sizes) rather than an equally spaced grid, and the culling table defines the priority of sounds that reach the listener when the sound source object and listener are located at the center of the voxel. Note that the voxels here do not necessarily have to be rectangular parallelepipeds.

[0223] The advantage of this method is that it takes into account the fact that, depending on the shape of the room and the arrangement of obstacles, there are places where the spatial resolution of the culling table can be set coarsely and places where it must be set finely. For example, if there are no walls or obstacles near the sound source object or listener, the spatial resolution of the culling table can be set coarse. On the other hand, if there are walls or obstacles near the sound source object or listener, the spatial resolution of the culling table can be set finely. By designing a culling table taking into account such non-uniformity in spatial resolution, it is possible to reduce the size of the culling table and determine the priority of sounds that reach the appropriate listener.

[0224] For convenience, FIG. 22 shows the positions of sound source objects and listeners on an XY plane, using voxel numbers (identifiers) assigned to voxels.

[0225] If the positions of the sound source object and listener are as shown in Figure 22, the sound source object is considered to be located at the center of voxel V0, and the listener is considered to be located at the center of voxel V17, and the culling table is referenced using (V0, V17) as an index to obtain the priority of the sound that will reach the listener.

[0226] An example of the culling table is shown in Fig. 23. In Fig. 23, the left column shows the identifiers of voxels at the positions of sound source objects, the center column shows the identifiers of voxels at the positions of listeners, and the right column shows the priority corresponding to the combination of the coordinate position of the sound source object converted by the voxel identifier and the coordinate position of the listener.

[0227] As shown in FIG. 23, the priority of direct sound, reverberant sound, reflected sound, and diffracted sound is determined by the combination of the voxel number where the sound source object is located and the voxel number where the listener is located, and information on that priority is obtained.

[0228] Referring now to Figure 24, Figure 24 shows a block diagram of another configuration of a decoder that includes a rendering unit 2420 instead of the rendering unit 1520. This configuration is characterized in that a direct sound generation unit 1521, a reverberation sound generation unit 1522, a reflected sound generation unit 1523, and a diffracted sound generation unit 1524 are connected in series. With this configuration, the sound generated by the generation unit in the previous stage affects the generation unit in the current stage, making it possible to provide accurate immersive audio that is closer to actual spatial audio.

[0229] A culling table is referenced based on the position of the sound source object and the position of the listener included in the metadata, and the priority of the sounds to be delivered to the listener is obtained. According to this priority, input switch units SW11, SW21, SW31, and SW41 and output switch units SW12, SW22, SW32, and SW42 are controlled to generate the sounds to be delivered to the listener. The input switch units SW11, SW21, SW31, and SW41 are all configured like the input switch unit SW1 shown in Figure 25, and the output switch units SW12, SW22, SW32, and SW42 are all configured like the output switch unit SW2 shown in Figure 26.

[0230] Specifically, input switch section SW11 in the preceding stage of direct sound generation section 1521 and output switch section SW12 in the following stage, input switch section SW21 in the preceding stage of reverberation sound generation section 1522 and output switch section SW22 in the following stage, input switch section SW31 in the preceding stage of reflected sound generation section 1523 and output switch section SW32 in the following stage, and input switch section SW41 in the preceding stage of diffracted sound generation section 1524 and output switch section SW42 in the following stage synchronize with each other using control information provided by control information determination section 1512, and control whether or not to operate direct sound generation section 1521, reverberation sound generation section 1522, reflected sound generation section 1523, and diffracted sound generation section 1524.

[0231] 27 will be described. In the above example, a control was described in which the switches for a predetermined number of generation units with the highest priority obtained from the culling table are turned on and the switches for the other generation units are turned off, but this example is characterized in that the switches for the generation units with the highest priority that fit within a predetermined amount of calculation are turned on and the switches for the other generation units are turned off.

[0232] In this example, the maximum amount of sound that reaches the listener can be preserved while maintaining a ceiling that ensures that the decoder's computational load does not exceed a predetermined value. Therefore, even when the decoder is incorporated into a device with limited processing power, stable operation is possible, and it is possible to provide the highest quality immersive audio possible within that processing power.

[0233] The figure shows a flowchart illustrating the operation of the decoder in this example. According to the priority given by the culling table reference unit 1511, the control information determination unit 1512 initializes the amount of calculation (S2701) and then accumulates (sums up) the amount of calculation for the generator with the highest priority (S2702). Each time the control information is accumulated, it compares this accumulated value with a predetermined threshold (S2703). If the accumulated value is smaller than the threshold (No in S2703), it accumulates the amount of calculation for the generator with the next highest priority (returns to S2702 via S2706). If the accumulated value is larger than the threshold (Yes in S2703), it sets and generates control information so that the generator with a higher priority than the most recently accumulated value is turned on and the other generators are turned off (S2704). The generated control information is then output (S2705).

[0234] 28 and 29 will be described. When a sound source object or a listener is not located on a three-dimensional grid point, the culling table is referenced on the assumption that the sound source object or the listener is located on the grid point closest to the sound source object or the listener. However, if the intervals between the grid points assumed in the culling table are large in order to reduce the amount of calculation required when designing the culling table, the actual position of the sound source object or the listener may differ from the position of the grid point, which may result in the problem that accurate priority information cannot be obtained.

[0235] To solve this problem, priority information may be obtained by using information on a plurality of lattice points located near the sound source object or the listener.

[0236] A specific method is explained below. Figure 28 shows the relationship between the position of the sound source object and each nearby lattice point, and Figure 29 shows the relationship between the position of the listener and each nearby lattice point. For simplicity, the explanation will be limited to the XY plane.

[0237] The position of the sound source object and each nearby grid point C ｉ The distance between ｉ (Fig. 28), the listener's position and each nearby grid point C ｊ The distance between ｊ (Fig. 29), the sound source object is at lattice point C ｉ and the listener is located at grid point C ｊ The probability P(C ｊ , C ｉ ) is defined as the following equation (1):

[0238]

[0239] Here, P(C ｊ ) and P(C ｉ ) is calculated from the distance between the position of the sound source object or the position of the listener and each grid point using the following equations (2) and (3).

[0240]

[0241]

[0242] When direct sound is A, reverberant sound is B, reflected sound is C, and diffracted sound is D, the evaluation value of each sound {E(X); X=A, B, C, D} is given by the following equation (4).

[0243]

[0244] Here, the function Order(C ｉ , C ｊ , X) is the point where the sound source object is located at the lattice point C ｉ , the listener is at grid point C ｊis a function that uses a culling table to return the priority of sound X that will reach the listener when the listener is located at , and the function W(n) represents a function that returns the weight when the priority is n. For example, the function W(n) is predefined so that the higher the priority, the greater the weight is given, such that W(0) = 10 when the priority is n = 0, W(1) = 5 when n = 1, W(2) = 2 when n = 2, and W(3) = 1 when n = 3.

[0245] The magnitudes of these evaluation values ​​{E(X); X=A, B, C, D} are compared, and the priority is determined in descending order of the evaluation value E(X).

[0246] 30 will be described. This example has the feature that the culling table is provided to the decoder as metadata when the decoder is initialized. This eliminates the need for the decoder to create a table when it is running, and reduces the amount of calculations required when the decoder is running.

[0247] A flowchart showing the operation in this example is shown in FIG.

[0248] First, the decoder is initialized (S3001), and at that time, metadata storing the culling table is received and the culling table is acquired (S3002). The acquired culling table is used in the subsequent operation of the decoder.

[0249] It is determined whether metadata is to be received (S3003). If metadata is not to be received (No in S3003), this process ends.

[0250] If metadata is received (Yes in S3003), the decoder operates (S3004) to perform acoustic processing based on the metadata, and generates and outputs immersive audio.

[0251] Then, the process returns to step S3003, where it is determined whether or not metadata has been received.

[0252] Here, the algorithm is defined based on whether or not metadata has been received, but it is not limited to this and the algorithm may also be defined based on whether or not a bit stream, multiplexed data, packet, etc. has been received.

[0253] 31 will be described. This example is characterized in that a culling table is provided to the decoder as metadata when the decoder is initialized, and if part or all of the culling table is included in the metadata while the decoder is running, part or all of the culling table is updated. This eliminates the need for the decoder to create a table during operation, and makes it possible to change the culling table to keep up with changes in spatial information, etc., thereby reducing the amount of calculation required when the decoder is running while maintaining the quality of immersive audio.

[0254] A flowchart showing the operation in this example is shown in Figure 31. Some of the operations are similar to those in Figure 30 above, and so the same reference numerals may be used.

[0255] The decoder is initialized (S3001), and at that time, metadata storing the culling table is received and the culling table is acquired (S3002). The acquired culling table is used in the subsequent operation of the decoder.

[0256] It is determined whether metadata is to be received (S3003). If metadata is not to be received (No in S3003), this process ends.

[0257] When metadata is received (Yes in S3003), it is determined whether the metadata includes a part or all of the culling table (S3101). If it does (Yes in S3101), a part or all of the culling table is updated (S3102).

[0258] The decoder performs acoustic processing based on this metadata (decoder operation (S3004)), and generates and outputs immersive audio.

[0259] Then, the process returns to step S3003, where it is determined whether or not metadata has been received.

[0260] Here, the algorithm is defined based on whether or not metadata has been received, but it is not limited to this and the algorithm may also be defined based on whether or not a bit stream, multiplexed data, packet, etc. has been received.

[0261] 32 will be described. While the rendering operation using a culling table when the number of sound source objects and listeners is one has been described above, here, the rendering operation using a culling table when the number of at least one of the sound source objects or listeners is multiple (two or more) is characterized in that a culling table suited to each combination of sound source objects and listeners is used. By using a culling table suited to the combination of sound source objects and listeners, more accurate priority information can be obtained than when a common culling table is used, and the operation of the means for generating sounds that are less necessary for the listeners can be accurately stopped.

[0262] FIG. 32 illustrates an example in which multiple objects 98 and 98a and listeners (users 99 and 99a) are arranged. For convenience, an XY plane is shown. In this figure, two sound source objects and two listeners are arranged. In this case, sounds reaching one listener (user 99) are emitted from sound source object 98 and sound source object 98a, respectively. Two culling tables are used: culling table (M) generated based on the positional relationship between sound source object 98 and one listener, and culling table (N) generated based on the positional relationship between sound source object 98a and one listener. The culling table (M) is used to determine the priority of sounds emitted from sound source object 98 when the sounds reach one listener, and the culling table (N) is used to determine the priority of sounds emitted from sound source object 98 when the sounds reach one listener.

[0263] Similarly, the sounds that reach the other listener (user 99a) are emitted from sound source object 98 and sound source object 98a, respectively, and two culling tables are used: culling table (O) generated from the positional relationship between sound source object 98 and the other listener, and culling table (P) generated from the positional relationship between sound source object 98a and the other listener. Culling table (O) is used to reference the priority of sounds when sounds emitted from sound source object 98 reach the other listener, and culling table (P) is used to reference the priority of sounds when sounds emitted from sound source object 98a reach the other listener.

[0264] 33 and 34 will be described. In the above examples, sounds with low priority were culled based on the priority obtained by referencing the culling table. In contrast, in this example, instead of culling low-priority sounds, multiple sounds that are subject to culling with low priority are integrated together and considered as virtual sounds output from a virtual object that is a virtual sound source object. This allows multiple sounds with low priority to be replaced with sounds output from a smaller number of virtual objects, making it possible to reduce the amount of calculation. In addition, because multiple sounds with low priority are not completely discarded, it is possible to avoid, to a certain extent, degradation of immersive audio due to culling.

[0265] Figure 33 shows a conceptual diagram of the culling process in this example. This diagram illustrates an example in which reverberant sounds (c) to (g) are selected as sounds with low priority. As shown in this diagram, multiple reverberant sounds reaching the listener are combined through integration processing into sounds output from a small number of virtual objects 96, as shown in Figure 34, and output toward the listener.

[0266] Here, the explanation has been given using reverberant sound as an example, but the present invention is not limited to this, and it is also possible to perform integration processing instead of culling processing on reflected sound and diffracted sound in the same way.

[0267] One method for combining multiple sounds with low priority is to add and integrate the sounds to be culled and regard the resulting sound as a sound output from the virtual object 96. When adding the sounds, at least one of the energy or phase of each sound may be adjusted before adding them.

[0268] In contrast to the first embodiment described so far, the priority order in the culling table may be determined through a two-stage process. Specifically, in the first stage, the priority order is provisionally determined using the culling table based on the listener position and the object position, as in the previous cases. Next, in the second stage, the sound generation units with low priority order are analyzed, and the priority order is revised (in other words, corrected) based on the analysis results to determine the final priority order. For example, an index that takes into account the auditory characteristics of the listener may be used as this analysis method.

[0269] This makes it possible to improve performance degradation caused by a mismatch in the sound environment between when the culling table was designed and when it was in operation, for example.

[0270] Here, the sound generation unit refers to one or more of the direct sound generation unit 1521 that generates direct sound, the reverberation sound generation unit 1522 that generates reverberation sound, the reflected sound generation unit 1523 that generates reflected sound, and the diffracted sound generation unit 1524 that generates diffracted sound.

[0271] Up to this point, the invention has been described based on the propagation of sound, but the invention is not limited to sound propagation and can also be applied to, for example, light propagation.

[0272] Regarding light propagation, the present invention applies to computer graphics that generate scenes based on direct light, reflected light, and diffracted light. Specifically, the priority of direct light, reflected light, and diffracted light is referenced from a culling table based on the positional relationship between the light source and the user in a virtual space or a space that combines virtual space and real space. Based on this priority, the operation of the light generation unit for low-priority light is stopped to generate computer graphics. As a result, the degradation of the quality of the computer graphics presented to the user is minimal, and the amount of calculation required to generate computer graphics can be significantly reduced, as the generation of only low-priority light is stopped.

[0273] <Functional Description of Rendering Unit, Example 2> FIGS. 35 to 57 are diagrams for explaining a specific example of an audio reproduction system according to Example 2 of the embodiment.

[0274] In this second embodiment, when the shape of the room, the material of the walls (reflectivity, absorption rate, etc.), and the shape and material of the obstacles (same as above) are given, the combinations of sounds to be integrated from multiple sounds such as direct sound, reflected sound, reverberation sound, and diffracted sound for the possible positional relationship between the listener and the sound source object are determined in advance, and an integration table is created in advance as a table for integration processing.

[0275] FIG. 35 is a block diagram of a decoder according to this example (i.e., a rendering unit 3520, a sound generation unit 3530, and a spatial information management unit 3510). The basic idea of ​​this example is to refer to an integration table based on the position of a sound source object and the position of a listener, identify multiple sounds to be integrated at those positions, and use information on those multiple sounds to construct virtual objects that are fewer in number than the sounds to be integrated, thereby generating sounds that reach the listener. This reduces the number of sounds input to the sound generation unit 3530 at the subsequent stage, and reduces the amount of calculation.

[0276] First, metadata is provided to the decoder. The structure of the metadata is shown in Figure 16, and its explanation will be omitted here.

[0277] The integration table is referenced based on the object position and listener position contained in the metadata, and the identifier of the sound to be integrated among the sounds that reach the listener is obtained.

[0278] Direct sound generation unit 1521, reverberation sound generation unit 1522, reflected sound generation unit 1523, and diffracted sound generation unit 1524 each receive an audio signal and metadata, generate direct sound, reverberation sound, reflected sound, and diffracted sound, and output these to integrating unit 3521. Note that these audio signals may be generated by decoding encoded audio data included in the metadata using an audio data decoder (not shown) and provided to direct sound generation unit 1521, reverberation sound generation unit 1522, reflected sound generation unit 1523, and diffracted sound generation unit 1524, respectively, or audio data included in the metadata may be provided to direct sound generation unit 1521, reverberation sound generation unit 1522, reflected sound generation unit 1523, and diffracted sound generation unit 1524, respectively.

[0279] The integration unit 3521 integrates sounds corresponding to the identifiers acquired by the integration table reference unit 3511 from among the signals input to the integration unit 3521, and outputs the integrated sounds to the sound generation unit 3530. Sounds that do not correspond to the identifiers are output to the sound generation unit 3530 as they are.

[0280] The sound generation unit 3530 performs sound processing such as HRTF (Head related transfer function) on the signal input from the integration unit 3521 and outputs the result as an output signal. This sound processing is performed to suit the output format to the listener, such as headphones or multi-channel speakers, and the output signal is provided to the listener.

[0281] Now, let us consider Figures 36 and 37. These figures show diagrams similar to Figure 19. Figure 36 shows the sounds that reach the listener, including (a) direct sound, (b) reflected sound, (c) to (g) reverberation sounds, and (h) diffracted sound.

[0282] When the position of the sound source object and the position of the listener are as shown in the figure, in order to refer to the integrated table, the sound source object is considered to be located at the lattice point (3, 1) closest to the position in the figure, and similarly, the listener is considered to be located at the lattice point (5, 5) closest to the position in the figure.

[0283] Next, the integration table is referenced based on the lattice point (3,1) near the position of the sound source object and the lattice point (5,5) near the position of the listener, and the identifiers of the sounds to be integrated are obtained. Figure 36 shows an example in which the identifiers indicating the reverberation sounds (d) and (e) are obtained from the integration table.

[0284] Note that, as a method of integrating multiple sounds to be integrated to form a virtual object, for example, there is a method of adding the sounds to be integrated and treating them as a sound output from the virtual object. When adding, at least one of the energy or phase of each sound may be adjusted before adding them. Note that the method given here is merely an example, and the method of integrating multiple sounds is not limited to this method.

[0285] Furthermore, the position of the virtual object is either an area surrounded by the direction connecting the listener and one sound and the direction connecting the listener and the other sound (the hatched area in FIG. 38 ). Alternatively, the position of the virtual object may be either an area surrounded by a direction obtained by loosening the direction connecting the listener and one sound to the outside and a direction obtained by loosening the direction connecting the listener and the other sound to the outside (the hatched area in FIG. 39 ).

[0286] Next, a flowchart of the operation of the decoder in the above configuration is shown in FIG.

[0287] First, it is determined whether metadata is input (S4001). If metadata is input (Yes in S4001), the process proceeds to the generation of direct sound, and if metadata is not input (No in S4001), the process ends.

[0288] If metadata is input (Yes in S4001), the integration table is referenced based on the positions of the listener and sound source object (S4002) to obtain the identifier of the sound to be integrated.

[0289] Furthermore, the sounds that reach the listener, that is, direct sounds, reverberant sounds, reflected sounds, and diffracted sounds, are each generated (S4003 to S4006).

[0290] The sounds to be integrated, which are identified by the integration table, are integrated to form a virtual object, and the sound of the virtual object is generated (S4007).

[0291] As a result of the integration process, the remaining direct sound, reverberant sound, reflected sound, diffracted sound, and generated sound from the virtual object are subjected to stereophonic signal processing such as HRTF (Head related transfer function) to generate a stereophonic signal (S4008), which is output to a device used by the listener, such as headphones.

[0292] Then, the process returns to step S4001, where it is determined whether new metadata is to be input.

[0293] Next, Fig. 41 will be described. Fig. 41 shows an example of an integration table. In Fig. 41, the left column shows the coordinate position (X, Y, Z) of the sound source object, the center column shows the coordinate position (X, Y, Z) of the listener, and the right column shows the identifiers of the sounds to be integrated that correspond to the combination of the coordinate position of the sound source object and the coordinate position of the listener. The identifiers of the sounds to be integrated are shown as two or more identifiers enclosed in parentheses, and when there are multiple parentheses, it means that the sounds of the identifiers in the respective parentheses will be integrated into each other.

[0294] The identifiers of the sounds to be integrated can be obtained by referencing the integration table based on the positions of the sound source object and the listener. Figure 41 shows an example of the configuration of the integration table, which shows how identifiers of the sounds to be integrated are stored for all or some combinations of the positions of the sound source object and the listener on predetermined lattice points.

[0295] For example, in FIG. 41, if the position of the sound source object is (O ｘ（Ｌ-１）,O ｙ（Ｍ-１） ,O ｚ（Ｎ-１） ), the listener position is (L ｘ（０） ,L ｙ（０） ,L ｚ（０） ), the identifier (2, 10) to be integrated is acquired. Therefore, the sound with the identifier 2 and the sound with the identifier 10 are integrated to form one virtual object, and the sound is generated and output.

[0296] Also, if the position of the sound source object is (O ｘ（０） ,O ｙ（０） ,O ｚ（０） ), the listener position is (L ｘ（ｌ） ,L ｙ（ｍ） ,L ｚ（ｎ） ), identifiers (4, 8) and (14, 19) to be integrated are acquired. In this case, the sound with identifier 4 and the sound with identifier 8 are integrated, and the sound with identifier 14 and the sound with identifier 19 are integrated to form virtual objects, and the sounds are generated and output.

[0297] Also, if the object's position is (O ｘ（ｌ） ,O ｙ（ｍ） ,O ｚ（ｎ） ), the listener position is (L ｘ（ｌ） ,L ｙ（ｍ） ,L ｚ（ｎ） ), the identifiers (1, 12, 13, 14) to be integrated are acquired. Thus, the four sounds, the sound with the identifier 1, the sound with the identifier 12, the sound with the identifier 13, and the sound with the identifier 14, are integrated to form one virtual object, and the sound is generated and output.

[0298] This integrated table may be used to identify sounds to be culled. For example, if the position of a sound source object is (O ｘ（Ｌ-１） ,O ｙ（Ｍ-１） ,O ｚ（Ｎ-１） ), the listener position is (L ｘ（０） ,L ｙ（０） ,L ｚ（０）), the identifier (2, 10) to be integrated is obtained. When an integration table is used to identify the sounds to be culled, at least one of the sounds represented by the identifiers to be integrated is culled rather than integrated. When there are multiple parentheses, they are used so that at least one of the sounds of the identifiers in each parentheses is culled.

[0299] The integration table can be created in advance. Specifically, for example, the direct sound, reverberant sound, reflected sound, and diffracted sound that actually reach the listener may be calculated for all or some combinations of the positions of the sound source object and the listener position, and the sounds to be integrated may be determined based on an index that takes into account auditory characteristics, such as the importance of these signals, such as the magnitude of energy or energy corrected for auditory characteristics. Alternatively, the sounds to be integrated may be determined based on the intersection angle (angle formed) between two sounds when they reach the listener or the level ratio of the two sounds. Creating an integration table requires a large amount of calculation, but since it can be created offline and does not require consideration of real-time performance, it can be created in advance.

[0300] Alternatively, a content creator or a service provider may design and create a part or all of the integrated table.

[0301] The integrated table may be stored in the decoder in advance as the initial state of the decoder. In this case, it is possible to reduce the amount of calculation using the integrated table without any particular processing.

[0302] Alternatively, the unified table may be obtained during the decoder initialization process, or some or all of the unified table may be updated while the decoder is running, as will be discussed below.

[0303] Although the integrated table described here is designed based on both the position of the sound source object and the position of the listener, the integrated table may also be designed based on either the position of the sound source object or the position of the listener in order to reduce the amount of calculation required when designing the integrated table and the memory required for the integrated table.

[0304] Referring now to Figure 42, Figure 42 is a block diagram of another configuration of a decoder that includes a rendering unit 4220 instead of the rendering unit 3520. A feature of this configuration is that a direct sound generation unit 1521, a reverberation sound generation unit 1522, a reflected sound generation unit 1523, and a diffracted sound generation unit 1524 are connected in series. With this configuration, the sound generated by the generation unit in the previous stage can affect the generation unit in the current stage, making it possible to provide accurate immersive audio that is closer to actual spatial audio.

[0305] In this figure, all output signals from the diffracted sound generation unit 1524 are provided to the integration unit 3521, but this is not limited to this. Alternatively, a portion of the output signal from at least one of the direct sound generation unit 1521, the reverberation sound generation unit 1522, the reflected sound generation unit 1523, and the diffracted sound generation unit 1524 may not enter the integration unit 3521, but may be output to the direct sound generation unit 4221.

[0306] 43 will be described. In the above examples, as shown in Figures 36 and 37, the space in which the sound source objects and listeners exist is divided into grids, and the integrated table is defined on the premise that the sound source objects and listeners are located on arbitrary grid points.

[0307] Here, the three-dimensional space is divided not by a grid but by voxels of any size, and the integration table obtains the identifier of the sound to be integrated based on both the identifier of the voxel to which the object belongs and the identifier of the voxel to which the listener belongs. Note that the voxels here do not necessarily have to be rectangular parallelepipeds. The voxel configuration is as explained in Figure 22, so an explanation of it will be omitted here.

[0308] An example of the configuration of the integration table in this example is shown in Fig. 43. As shown in Fig. 43, the identifier of the sound to be integrated is obtained by combining the voxel number to which the sound source object belongs and the voxel number to which the listener belongs, and a virtual object is constructed from information on the multiple sounds indicated by the obtained identifier.

[0309] This integrated table may be used to identify sounds to be culled.

[0310] Now, let us consider Fig. 44. While the rendering operation using an integration table when the number of sound source objects and listeners is one has been described above, here, the rendering operation using an integration table when there are multiple (two or more) sound source objects or listeners is characterized in that an integration table suited to each combination of sound source objects and listeners is used. By using an integration table suited to each combination of sound source objects and listeners, more accurate information on the combination of sounds to be integrated can be obtained than by using a common integration table.

[0311] FIG. 44 illustrates an example in which multiple objects 98 and 98a and listeners (users 99 and 99a) are arranged. For convenience, an XY plane is shown. In this figure, two sound source objects and two listeners are arranged. In this case, the sounds reaching one listener (user 99) are emitted from sound source object 98 and sound source object 98a, respectively. Two integration tables are used: integration table (M) generated from the positional relationship between sound source object 98 and one listener; and integration table (N) generated from the positional relationship between sound source object 98a and one listener. Integration table (M) is used to obtain a combination of sounds to be integrated when sounds emitted from sound source object 98 reach one listener, and integration table (N) is used to obtain a combination of sounds to be integrated when sounds emitted from sound source object 98a reach one listener.

[0312] Similarly, the sounds that reach the other listener (user 99a) are emitted from sound source object 98 and sound source object 98a, respectively, and two integration tables are used: integration table (O) generated from the positional relationship between sound source object 98 and the other listener, and integration table (P) generated from the positional relationship between sound source object 98a and the other listener. Integration table (O) is used to obtain combinations of sounds that are to be integrated when sounds emitted from sound source object 98 reach the other listener, and integration table (P) is used to obtain combinations of sounds that are to be integrated when sounds emitted from sound source object 98a reach the other listener.

[0313] 45 will be described. This example is characterized in that the integrated table is provided to the decoder as metadata when the decoder is initialized. This eliminates the need for the decoder to create a table during operation, reducing the amount of calculation required when the decoder is running.

[0314] A flowchart of the operation in this example is shown in FIG.

[0315] First, the decoder is initialized (S4501), and at that time, metadata storing the unified table is received and the unified table is acquired (S4502). The acquired unified table is used in the subsequent operation of the decoder.

[0316] It is determined whether metadata is to be received (S4503). If metadata is not to be received (No in S4503), this processing ends.

[0317] If metadata is received (Yes in S4503), the decoder operates (S4504) to perform acoustic processing based on the metadata, and generates and outputs immersive audio.

[0318] Then, the process returns to step S4503, where it is determined whether or not metadata has been received.

[0319] Here, the flowchart is defined based on whether or not metadata has been received, but the flowchart may also be defined based on whether or not a bit stream, multiplexed data, packets, etc. have been received.

[0320] 46 will be described. This example is characterized in that the integrated table is provided to the decoder as metadata when the decoder is initialized, and if part or all of the integrated table is included in the metadata while the decoder is operating, part or all of the integrated table is updated. This eliminates the need for the decoder to create a table during operation, and enables the integrated table to keep up with changes in spatial information, etc., thereby reducing the amount of calculation required when the decoder is operating while maintaining the quality of immersive audio.

[0321] A flowchart showing the operation in this example is shown in Figure 46. Some of the operations are similar to those in Figure 45 above, and so the same reference numerals may be used.

[0322] The decoder is initialized (S4501), and at that time, the metadata storing the integrated table is received and the integrated table is acquired (S4502). The acquired integrated table is used in the subsequent operation of the decoder.

[0323] It is determined whether metadata is to be received (S4503). If metadata is not to be received (No in S4503), this processing ends.

[0324] When metadata is received (Yes in S4503), it is determined whether the metadata includes a part or all of the integrated table (S4601). If it does (Yes in S4601), a part or all of the integrated table is updated (S4602).

[0325] The decoder performs acoustic processing based on this metadata (decoder operation (S4504)), generating and outputting immersive audio.

[0326] Then, the process returns to step S4503, where it is determined whether or not metadata has been received.

[0327] Here, the flowchart is defined based on whether or not metadata has been received, but the flowchart may also be defined based on whether or not a bit stream, multiplexed data, packets, etc. have been received.

[0328] Reference is now made to Figures 47 and 48, which show a diagram similar to Figure 36.

[0329] In Figure 47, the reverberation sounds of (d) and (e) are selected as the sounds to be integrated. At this time, if the above explanation is followed, as shown in the figure, the reverberation sounds of (d) and (e) are integrated to form a virtual object, and sound is output from the virtual object toward the listener. On the other hand, in this example, in addition to the reverberation sounds of (d) and (e), the reverberation sounds of (c) and (f), which are nearby, are also used to form a virtual object, and sound is output toward the listener.

[0330] The reason why the virtual object is configured to include not only the sounds selected as the targets of integration but also nearby sounds is to prevent changes in the sounds selected as the targets of integration when the sound source object or the listener moves over time (for example, a change from the reverberation of (d) and the reverberation of (e) to the reverberation of (e) and the reverberation of (f)), which would cause the sound output from the virtual object to change (from a virtual object composed of the reverberation of (d) and the reverberation of (e) to a virtual object composed of the reverberation of (e) and the reverberation of (f)). When such a change in the virtual object occurs, the position where the sound is generated or the characteristics of the generated sound may suddenly change, causing the listener to perceive a deterioration in the quality of the immersive audio.

[0331] As in this example, by configuring a virtual object including not only the sound selected as the target of integration but also nearby sounds, even if the sound source object or the listener moves, the virtual object will be configured to include sounds that will be selected at the destination because the range of sounds to be integrated is wide. Therefore, changes to the virtual object are less likely to occur, which has the effect of reducing the frequency of the degradation in immersive audio quality described above.

[0332] Please refer to Figures 49 to 53. Figures 49 to 52 show views similar to Figure 36.

[0333] In this example, before the listener moves, the reverberation of (d) and the reverberation of (e) are selected as the sounds to be integrated (FIG. 49). Then, after a certain time has passed and the listener moves, the reverberation of (e) and the reverberation of (f) are selected (FIG. 50).

[0334] Before the listener moves, a virtual object is constructed based on the reverberation sound (d) and the reverberation sound (e) (FIG. 51), and the sound generated by the virtual object is output to the listener. On the other hand, after the listener moves, a virtual object is constructed based on the reverberation sound (e) and the reverberation sound (f) (FIG. 52), and the sound generated by the virtual object is output to the listener.

[0335] At this time, changes occur in the position and sound characteristics of the reverberation sounds (e) and (f) generated by the virtual object, and the listener may feel as if the position and characteristics of the sound have suddenly changed, resulting in a perception of a deterioration in the quality of the immersive audio.

[0336] To solve this problem, we use windowing and addition to process the sound so that its position and characteristics change gradually, thereby mitigating the degradation of immersive audio quality.

[0337] Specifically, the end of the virtual sound obtained by combining the reverberation sounds (d) and (e) generated by the virtual object before the listener moved and the beginning of the virtual sound obtained by combining the reverberation sounds (e) and (f) generated by the virtual object after the listener moved are generated so as to overlap in time, and these are multiplied by the corresponding window functions and added (FIG. 53), thereby generating the sound that is finally output to the listener. Here, the window function for the virtual sound obtained by combining the reverberation sounds (d) and (e) has a shape that gradually attenuates at the connection portion, and the window function for the virtual sound obtained by combining the reverberation sounds (e) and (f) has a shape that gradually amplifies at the connection portion.

[0338] Furthermore, the position of the virtual object is controlled so that it changes gradually from the position of the virtual object before the listener moves to the position of the virtual object after the listener moves (FIG. 52).

[0339] By implementing this process, changes in the position and characteristics of the sound are smoothed, preventing degradation of the quality of the immersive audio, ultimately providing high-quality immersive audio to the listener.

[0340] Reference is now made to Figures 54 to 57, which show views similar to Figure 36.

[0341] 54 to 57 show cases where the device in which the decoder is implemented has a large storage capacity and high processing power (FIGS. 54 and 55, hereafter referred to as a high-performance device (HP device)), and cases where the device in which the decoder is implemented has a small storage capacity and low processing power (FIGS. 56 and 57, hereafter referred to as a low-performance (LP device)). The lattice spacing is set narrower in an HP device, while the lattice spacing is set wider in an LP device; the narrower the lattice spacing, the larger the storage capacity and the higher the processing power required, but the more accurately the sounds to be integrated can be determined, making it possible to generate high-quality immersive audio.

[0342] In the HP device, the integration table is referenced using the positions of the sound source object and listener as arguments, and as a result, the reverberation sounds (d) and (e) are selected (dashed circle in Figure 54). Therefore, a virtual object is constructed based on the reverberation sounds (d) and (e), and sound is generated and output to the listener (Figure 55). Because the grid spacing is set narrow in the HP device, the number of combinations of the positions of the sound source object and listener is large, the size of the integration table is large, and the amount of calculation increases. However, the accuracy of the integration table is improved, and it is possible to accurately select the sounds to be integrated.

[0343] On the other hand, with an LP device, the grid spacing is set wide, so the number of combinations of the sound source object and the listener's positions is reduced, the size of the integration table is small, and the amount of calculation is also reduced. However, the accuracy of the integration table is compromised, and it is not always possible to accurately select the sounds to be integrated. When using an LP device, unlike with an HP device, the reverberation sound (c) and the reverberation sound (d) are selected (dashed circle in Figure 56), and a virtual object is constructed from these two sounds, and the sound is output toward the listener (Figure 57).

[0344] Information about the device used by the listener is sent to a service provider or similar entity via communication or the like at the initialization stage when the listener receives an immersive audio service. The service provider will use this information about the device used by the listener to provide an immersive audio service based on an appropriate grid spacing.

[0345] In contrast to the second embodiment described so far, the identifiers of the sounds to be integrated in the integration table may be determined through a two-stage process. Specifically, in the first stage, the identifiers of the sounds to be integrated are provisionally determined using the integration table based on the listener position and the object position, as in the previous cases. Next, in the second stage, the provisionally determined sounds are analyzed, and the identifiers of the sounds to be integrated are modified (in other words, corrected) based on the analysis results to obtain the final sound identifiers. For example, an index that takes into account the auditory characteristics of the listener may be used as this analysis method.

[0346] This makes it possible to improve performance degradation caused by a mismatch in the sound environment between when the integrated table was designed and when it is in operation, for example.

[0347] Here, the sound generation unit refers to one or more of the direct sound generation unit 1521 that generates direct sound, the reverberation sound generation unit 1522 that generates reverberation sound, the reflected sound generation unit 1523 that generates reflected sound, and the diffracted sound generation unit 1524 that generates diffracted sound.

[0348] Furthermore, in the above examples, examples of generating direct sound, reflected sound, reverberant sound, and diffracted sound have been given, but this is not limited to these, and the present invention can be applied to any type of sound, regardless of its name or sound characteristics, as long as it is direct sound or a sound derived from direct sound that reaches the listener.

[0349] Also, a method for achieving hardware scalability by controlling the grid spacing has been described. However, this is not limiting, and hardware scalability may be achieved, for example, by controlling the size of a voxel according to the storage capacity and processing power of the device used by the listener.

[0350] Up to this point, the invention has been described based on the propagation of sound, but the invention is not limited to sound propagation and can also be applied to, for example, light propagation.

[0351] Regarding light propagation, the present invention is applicable to computer graphics that generate scenes based on direct light, reflected light, and diffracted light. Specifically, the light to be integrated is identified by referencing an integration table based on the positional relationship between the light source and the user in a virtual space or a space that combines virtual space and real space. The identified light is then used to construct a virtual object and output new light to generate computer graphics. This significantly reduces the amount of calculation required to generate computer graphics while minimizing degradation in the quality of the computer graphics.

[0352] <Functional Description of Rendering Unit, Example 3> FIGS. 58 to 70 are diagrams for explaining a specific example of an audio reproduction system according to Example 3 of the embodiment.

[0353] In the third embodiment, when spatial information (room shape, wall material (reflectivity, absorption rate, etc.), obstacle shape, material (same as left), etc.) is given, it is determined in advance whether or not each of a plurality of sounds such as direct sound, reflected sound, reverberation sound, and diffracted sound will reach the listener for the possible positional relationship between the listener and the sound source object, and an operation / non-operation table is created in advance that can refer to information on whether each of the direct sound generation unit 1521, reflected sound generation unit 1522, reverberation sound generation unit 1523, and diffracted sound generation unit 1524 will operate, using the positional information of the objects and listener arranged on the lattice points in the space as arguments.

[0354] FIG. 58 will be described. FIG. 58 is a block diagram of a decoder (i.e., a rendering unit 1520 and a spatial information management unit 5810) according to the third embodiment. The basic idea of ​​this embodiment is to refer to an action / non-action table based on the position of the sound source object and the position of the listener, identify action / non-action information indicating whether or not to perform an action to generate a sound that reaches the listener at those positions, and generate the sound that reaches the listener according to the action / non-action information. This makes it possible to cancel in advance the generation process of sounds that are less important among the sounds that reach the listener, thereby reducing the amount of calculation.

[0355] First, metadata is provided to the decoder. The structure of the metadata is shown in Figure 16, and its explanation will be omitted here.

[0356] The operation / non-operation table is referenced based on the object position and listener position included in the metadata, and operation / non-operation information for the sound that will reach the listener is obtained. This operation / non-operation information indicates whether the direct sound generation unit 1521, the reverberation sound generation unit 1522, the reflected sound generation unit 1523, and the diffracted sound generation unit 1524 should be operated or not, and the sound that will reach the listener is generated according to this information.

[0357] The metadata is provided to each of the direct sound generation unit 1521, reverberation sound generation unit 1522, reflected sound generation unit 1523, and diffracted sound generation unit 1524 via the switch unit 1513. The switch unit 1513 controls each generation unit in accordance with the operation / non-operation information acquired by referring to the operation / non-operation table.

[0358] Furthermore, audio signals are provided to direct sound generation unit 1521, reverberation sound generation unit 1522, reflected sound generation unit 1523, and diffracted sound generation unit 1524. The audio signals may be generated by decoding encoded audio data included in the input data using an audio data decoder (not shown), and provided to direct sound generation unit 1521, reverberation sound generation unit 1522, reflected sound generation unit 1523, and diffracted sound generation unit 1524, respectively, or audio data included in the input data may be provided to direct sound generation unit 1521, reverberation sound generation unit 1522, reflected sound generation unit 1523, and diffracted sound generation unit 1524, respectively.

[0359] Fig. 59 will be described. Fig. 59 is a configuration diagram of the switch unit. Switch unit 1513 here differs from switch unit 1513 described in Fig. 17 in that it uses operation / non-operation information instead of control information, but is otherwise similar.

[0360] When metadata is input and a switch is to be turned on, the metadata is provided to a generator that turns on the switch. Whether the switch is turned on or off is determined based on the operation / non-operation information. That is, if the operation / non-operation information indicates operation, the switch is turned on, and if the operation / non-operation information indicates non-operation, the switch is turned off.

[0361] The generation unit whose switch is turned on receives the metadata, generates one of direct sound, reverberant sound, reflected sound, or diffracted sound, and outputs it to the sound generation unit 1525. When the switch is turned off, the metadata is not sent to that generation unit, and the generation unit does not operate.

[0362] The audio generator 1525 performs audio processing such as HRTF (Head Related Transfer Function) on the signals output from each generator, and outputs the resulting signals as output signals. This audio processing is performed to suit the output format to the listener, such as headphones or multi-channel speakers, and the output signals are provided to the listener.

[0363] Although the decoder has the generators configured in parallel, the configuration is not limited to this, and the generators may be configured in series, which will be explained later.

[0364] Next, a flowchart of the operation of the decoder in the above configuration is shown in FIG.

[0365] First, it is determined whether metadata has been input (S1801). If metadata has been input (Yes in S1801), the operation / non-operation table is referenced (S6001). If metadata has not been input (No in S1801), the process ends.

[0366] Based on the positions of the listener and sound source object, the operation / non-operation table is referenced (S6001), operation / non-operation information for each generation unit is obtained, and based on this information, it is determined whether the switch for each generation unit is on or off.

[0367] Next, the operation / non-operation information for the direct sound generation unit 1521 is referenced (S1804), and if this information indicates operation (Yes in S1804), the switch is turned on and direct sound is generated (S1805). If the operation / non-operation information indicates non-operation (No in S1804), the switch is turned off and direct sound is not generated (S1805 is skipped).

[0368] Similarly, operation / non-operation information for the reverberation sound generation unit 1522 is referenced (S1806), and if this information indicates operation (Yes in S1806), the switch is turned on to generate reverberation sound (S1807), and if this information indicates non-operation (No in S1806), the switch is turned off to not generate reverberation sound (S1807 is skipped). Also, operation / non-operation information for the reflected sound generation unit 1523 is referenced (S1808), and if this information indicates operation (Yes in S1808), the switch is turned on to generate reflected sound (S1809), and if this information indicates non-operation (No in S1808), the switch is turned off to not generate reflected sound (S1809 is skipped). In addition, the operation / non-operation information for the diffracted sound generation unit 1524 is referenced (S1810), and if this information indicates operation (Yes in S1810), the switch is turned on and diffracted sound is generated (S1811), and if this information indicates non-operation (No in S1810), the switch is turned off and diffracted sound is not generated (S1811 is skipped).

[0369] Finally, the generated signals are subjected to stereophonic signal processing such as convolution processing of head-related transfer functions to generate stereophonic signals (S1812), which are output to a device used by the listener, such as headphones.

[0370] Then, the process returns to step S1801, where it is determined whether new metadata is to be input.

[0371] Referring to Fig. 61, Fig. 61 shows a conceptual diagram of an embodiment using an action / non-action table.

[0372] The space in which the sound source object and listener exist is divided into three-dimensional grids, and the operation / non-operation table stores operation / non-operation information for each generation unit that generates the sound that reaches the listener when the sound source object and listener are located on any grid point.

[0373] An example will be explained using Figure 61. For convenience, Figure 61 shows an XY plane, but in reality, the virtual space is expressed as a 3D grid. When the position of the sound source object and the position of the listener are given, the operation / non-operation table is referenced to obtain operation / non-operation information for each generator that generates the sound that will reach the listener (direct sound (a) in the figure, reverberant sounds (c) to (g) , reflected sound (b), and diffracted sound (h)).

[0374] If the sound source object is at the position in the figure and the listener is at position (A) in the figure, the listener can hear direct sound, reflected sound, reverberant sound, and diffracted sound, so the operating / non-operating information for this positional relationship is (direct sound, reflected sound, reverberant sound, diffracted sound) = (O, O, O, O). Here, O (circle) indicates that the generator is operating and generating sound that reaches the listener, and × (cross) indicates that the generator is not operating. Note that even if sound reaches the listener, if the sound level is below the loudness threshold, it is considered not to have reached the listener, and the generator is marked as not operating (×).

[0375] If either or both of the sound source object and the listener are not located on a three-dimensional grid point, the operation / non-operation table is referenced, assuming that either or both of the sound source object and the listener are located on the grid point closest to that position.

[0376] Fig. 62A will be described. Fig. 62A shows the configuration of the action / non-action table. In Fig. 62A, the left column shows the coordinate position (X, Y, Z) of the sound source object, the center column shows the coordinate position (X, Y, Z) of the listener, and the right column shows the action / non-action information corresponding to the combination of the coordinate position of the sound source object and the coordinate position of the listener. The circles and crosses in the action / non-action information are as already described in the explanation of Fig. 61.

[0377] By referring to the motion / non-motion table based on the position of the sound source object and the position of the listener, motion / non-motion information can be obtained. For example, in FIG. 62A, when the object position is (O ｘ（ｌ） ,O ｙ（ｍ） ,O ｚ（ｎ） ), the listener position is (L ｘ（ｌ） ,L ｙ（ｍ） ,L ｚ（ｎ） ), the operation / non-operation information (direct sound, reverberant sound, reflected sound, diffracted sound) is acquired as (○, ○, ×, ○). In other words, the operation / non-operation information obtained is that direct sound generation is operating, reverberant sound generation is operating, reflected sound generation is not operating, and diffracted sound generation is operating.

[0378] The operation / non-operation table can be created in advance. A specific method for this is to actually calculate the direct sound, reverberation sound, reflection sound, and diffracted sound that reach the listener for all combinations of the position of the sound source object and the position of the listener, calculate an index that takes into account the auditory characteristics, such as the importance of these signals, for example, the magnitude of energy or energy corrected by auditory characteristics, and set the device to operate if the index exceeds a reference threshold, and to not operate if it does not. This requires a huge amount of calculation, but because the operation / non-operation table can be created in advance, it is possible to realize this without having to consider limitations in the hardware processing power or real-time performance.

[0379] Alternatively, the content creator may design and create part or all of the action / non-action table.

[0380] The operation / non-operation table may be stored in the decoder in advance as the initial state of the decoder. In this case, it is possible to reduce the amount of calculation using the operation / non-operation table without any particular processing.

[0381] Alternatively, the operation / non-operation table may be obtained during the decoder initialization process, or some or all of the operation / non-operation table may be updated while the decoder is in operation, as will be described later.

[0382] Note that although the description here is of an integrated table designed based on both the position of the sound source object and the position of the listener, in order to reduce the amount of calculation required when designing the operation / non-operation table and the memory required for the operation / non-operation table, the operation / non-operation table may also be designed based on either the position of the sound source object or the position of the listener.

[0383] Fig. 62B will be described. Fig. 62B shows an action / non-action table according to another embodiment. The action / non-action table in Fig. 62A only indicates whether each generator is active or inactive based on the position of the sound source object and the position of the listener. The action / non-action table described in this figure is characterized in that it indicates whether each generator is active or inactive based on the position of the sound source object and the position of the listener, and also indicates the priority of the generator that generates the sound when an action is indicated.

[0384] This embodiment has the following advantages. By prioritizing the generators that indicate their operation, it becomes possible to disable the operation of generators with lower priorities until the computational load is reduced to a predetermined level. This prevents degradation of immersive audio quality, further reducing the computational load and facilitating implementation on devices with computational load limitations.

[0385] For example, in FIG. 62B, if the object position is (O ｘ（ｌ） ,O ｙ（ｍ） ,O ｚ（ｎ） ), the listener position is (L ｘ（ｌ） ,L ｙ（ｍ） ,L ｚ（ｎ）), (direct sound, reverberant sound, reflected sound, diffracted sound) = (2, 1, × , 3) ​​is acquired as the motion / non-motion information.

[0386] That is, the operation / non-operation information obtained indicates that direct sound operates but has a priority of 2, reverberant sound operates but has a priority of 1, reflected sound does not operate, and diffracted sound operates but has a priority of 3. Using this operation / non-operation information, it is determined whether each generation unit operates or not. The procedure is as follows.

[0387] A generation unit whose operation / non-operation information indicates non-operation is not operated. Furthermore, in addition to the operation amount of a generation unit that indicates non-operation, the operation amount of a generation unit that has the lowest priority is accumulated, and when the accumulated value exceeds a predetermined threshold, the generation unit that was the target of accumulation immediately before the threshold was exceeded is not operated. Only the remaining generation units are allowed to operate.

[0388] Fig. 63 will be described. Fig. 63 shows an operation / non-operation table according to another embodiment. A feature of this embodiment is that the operation / non-operation table is constructed by dividing reflected sounds into primary reflected sounds that hit a wall or obstacle once and reach the listener, secondary reflected sounds that hit a wall or obstacle twice and reach the listener, and higher-order reflected sounds that hit a wall or obstacle three or more times and reach the listener.

[0389] After direct sound, reflected sound has more energy than other sounds that reach the listener, such as reverberation and diffraction, so constructing an operation / non-operation table that takes into account sounds of secondary reflection and above will enable the selection of a more appropriate generation unit that will operate or not operate.

[0390] In Fig. 63, the position of the sound source object and the position of the listener are referenced to acquire the action / non-action information for each sound that reaches the listener. The sounds that reach the listener are direct sound, reverberation sound, first-order reflected sound, second-order reflected sound, higher-order reflected sound, and diffracted sound, and are the sounds for which action / non-action information is acquired.

[0391] For example, in FIG. 63, if the object position is (O ｘ（ｌ） ,O ｙ（ｍ） ,O ｚ（ｎ） ), the listener position is (L ｘ（Ｌ-１） ,Lｙ（Ｍ-１） ,L ｚ（Ｎ-１） ), the motion / non-motion information obtained is (direct sound, reverberation, first reflected sound, second reflected sound, higher reflected sound, diffracted sound) = (×, 〇, 〇, ×, ×, 〇).

[0392] In other words, the operation / non-operation information obtained is that the generation of direct sound is non-operational, the generation of reverberant sound is operating, the generation of primary reflected sound is operating, the generation of secondary reflected sound is non-operational, the generation of higher-order reflected sound is non-operational, and the generation of diffracted sound is operating.

[0393] Now, let us consider Fig. 64. In the above example, as shown in Fig. 61, the space in which the sound source objects and listeners exist is divided into equal-spaced grids in a three-dimensional space, and the action / non-action table is defined on the premise that the sound source objects and listeners are positioned at arbitrary grid points.

[0394] Here, the three-dimensional space is divided into cubes (voxels) of arbitrary size, rather than an evenly spaced grid, and the action / non-action table specifies the action / non-action information for the sound that reaches the listener when the sound source object and listener are located at the center of the voxel. Note that the voxels here do not necessarily have to be cubes; they can also be rectangular parallelepipeds. The voxel configuration is as explained in Figure 22, so a detailed explanation will be omitted here.

[0395] An example of the configuration of the action / non-action table in this example is shown in Fig. 64. As shown in Fig. 64, the action / non-action information of direct sound, reverberation sound, reflected sound, and diffracted sound is determined by the combination of the voxel number where the sound source object is located and the voxel number where the listener is located, and the action / non-action information is obtained.

[0396] Referring now to Figure 65, Figure 65 is a block diagram of another configuration of a decoder that includes a rendering unit 2420 instead of the rendering unit 1520. This configuration is characterized in that a direct sound generation unit 1521, a reverberation sound generation unit 1522, a reflected sound generation unit 1523, and a diffracted sound generation unit 1524 are connected in series. With this configuration, the sound generated by the generation unit in the previous stage affects the generation unit in the current stage, making it possible to provide accurate immersive audio that is closer to actual spatial audio.

[0397] Based on the object position and listener position included in the metadata, an operation / non-operation table is referenced to obtain operation / non-operation information for the sound that will reach the listener. According to this operation / non-operation information, input switch units SW11, SW21, SW31, and SW41 and output switch units SW12, SW22, SW32, and SW42 are controlled to generate the sound that will reach the listener. The input switch units SW11, SW21, SW31, and SW41 are all configured like the input switch unit SW1 shown in Figure 66, and the output switch units SW12, SW22, SW32, and SW42 are all configured like the output switch unit SW2 shown in Figure 67.

[0398] Specifically, input switch section SW11 in the preceding stage of direct sound generation section 1521 and output switch section SW12 in the following stage, input switch section SW21 in the preceding stage of reverberation sound generation section 1522 and output switch section SW22 in the following stage, input switch section SW31 in the preceding stage of reflected sound generation section 1523 and output switch section SW32 in the following stage, and input switch section SW41 in the preceding stage of diffracted sound generation section 1524 and output switch section SW42 in the following stage synchronize with each other using control information provided by control information determination section 1512, and control whether or not to operate direct sound generation section 1521, reverberation sound generation section 1522, reflected sound generation section 1523, and diffracted sound generation section 1524.

[0399] In this third embodiment, as explained in Figures 28 and 29 of the first embodiment, priority information may also be obtained by using information on multiple lattice points located in the vicinity of the sound source object or the listener.

[0400] 68 will be described. This example is characterized in that the operation / non-operation table is provided to the decoder as metadata when the decoder is initialized. This eliminates the need for the decoder to create a table when it is operating, and reduces the amount of calculation required when the decoder is operating.

[0401] A flowchart showing the operation in this example is shown in FIG.

[0402] The decoder is initialized (S6801), and at that time, metadata storing the operation / non-operation table is received and the operation / non-operation table is acquired (S6802). The acquired operation / non-operation table is used in the subsequent operation of the decoder.

[0403] It is determined whether metadata is to be received (S6803). If metadata is not to be received (No in S6803), this processing ends.

[0404] If metadata is received (Yes in S6803), the decoder operates (S6804) to perform acoustic processing based on the metadata, and generates and outputs immersive audio.

[0405] Then, the process returns to step S6803, where it is determined whether or not metadata has been received.

[0406] While the algorithm is defined here based on whether metadata has been received, the algorithm may also be defined based on whether a bitstream, multiplexed data, packets, etc. have been received. Referring to FIG. 69, this example is characterized in that the operation / non-operation table is provided to the decoder as metadata when the decoder is initialized, and if part or all of the operation / non-operation table is included in the metadata while the decoder is operating, part or all of the operation / non-operation table is updated. This eliminates the need for the decoder to create a table during operation, and enables the operation / non-operation table to keep up with changes in spatial information, etc., thereby reducing the amount of calculation required when the decoder is operating while maintaining the quality of immersive audio.

[0407] A flowchart showing the operation in this example is shown in Figure 69. Some of the operations are similar to those in Figure 68 above, and so the same reference numerals may be used.

[0408] The decoder is initialized (S6801), and at that time, metadata storing the operation / non-operation table is received and the operation / non-operation table is acquired (S6802). The acquired operation / non-operation table is used in the subsequent operation of the decoder.

[0409] It is determined whether metadata is to be received (S6803). If metadata is not to be received (No in S6803), this processing ends.

[0410] When metadata is received (Yes in S6803), it is determined whether the metadata includes a part or all of the operation / non-operation table (S6901). If it does (Yes in S6901), a part or all of the operation / non-operation table is updated (S6902).

[0411] The decoder performs acoustic processing based on this metadata (decoder operation (S6804)), generating and outputting immersive audio.

[0412] Then, the process returns to step S6803, where it is determined whether or not metadata has been received.

[0413] Here, the algorithm is defined based on whether or not metadata has been received, but it is not limited to this and the algorithm may also be defined based on whether or not a bit stream, multiplexed data, packet, etc. has been received.

[0414] 70 will be explained. While the rendering operation using the operation / non-operation table when the number of sound source objects and listeners is one has been explained above, here, the rendering operation using the operation / non-operation table when the number of at least one of the sound source objects or listeners is multiple (two or more) is characterized in that an operation / non-operation table suited to each combination of sound source object and listener is used. By using an operation / non-operation table suited to the combination of sound source object and listener, more accurate operation / non-operation information can be obtained than when a common operation / non-operation table is used, and the operation of sound generation means that is less necessary for the listener can be accurately stopped.

[0415] FIG. 70 illustrates an example in which multiple objects 98 and 98a and listeners (users 99 and 99a) are arranged. For convenience, the XY plane is shown. In this figure, two sound source objects and two listeners are arranged. In this case, the sounds reaching one listener (user 99) are emitted from sound source object 98 and sound source object 98a, respectively. Two action / non-action tables are used: an (M) action / non-action table generated based on the positional relationship between sound source object 98 and one listener, and an (N) action / non-action table generated based on the positional relationship between sound source object 98a and one listener. The (M) action / non-action table is used to reference the priority of the sound emitted from sound source object 98 when it reaches one listener, and the (N) action / non-action table is used to reference the priority of the sound emitted from sound source object 98a when it reaches one listener.

[0416] Similarly, the sounds that reach the other listener (user 99a) are emitted from sound source object 98 and sound source object 98a, respectively, and two action / non-action tables are used: action / non-action table (O) generated from the positional relationship between sound source object 98 and the other listener, and culling table (P) generated from the positional relationship between sound source object 98a and the other listener. The action / non-action table (O) is used to refer to the priority of sounds when sounds emitted from sound source object 98 reach the other listener, and the action / non-action table (P) is used to refer to the priority of sounds when sounds emitted from sound source object 98a reach the other listener.

[0417] In contrast to the third embodiment described above, the action / non-action information in the action / non-action table may be determined through a two-stage process. Specifically, in the first stage, the action / non-action table is used to provisionally determine which sound generation units are to be active and which are to be inactive, based on the listener position and the object position, as in the previous stage. Next, in the second stage, the sound of the active sound generation unit or the sound of the inactive sound generation unit is analyzed, and the action / non-action information is corrected (in other words, amended) based on the analysis results to obtain the final action / non-action information. For example, an index that takes into account the auditory characteristics of the listener may be used as an analysis method.

[0418] This makes it possible to improve the performance degradation caused by a mismatch in the sound environment between when the operation / non-operation table was designed and when it was in operation, for example.

[0419] Here, the sound generation unit refers to one or more of the direct sound generation unit 1521 that generates direct sound, the reverberation sound generation unit 1522 that generates reverberation sound, the reflected sound generation unit 1523 that generates reflected sound, and the diffracted sound generation unit 1524 that generates diffracted sound.

[0420] Up to this point, the invention has been described based on the propagation of sound, but the invention is not limited to sound propagation and can also be applied to, for example, light propagation.

[0421] Regarding light propagation, the present invention is applicable to computer graphics that generate scenes based on direct light, reflected light, and diffracted light. Specifically, a motion / non-motion table is created by simulating in advance whether direct light, reflected light, and diffracted light will reach the user based on the positional relationship between the light source and the user in a virtual space or a space that combines virtual space and real space. The motion / non-motion information obtained is tabulated and used to generate the computer graphics. During the computer graphics generation stage, the motion / non-motion information is referenced from the motion / non-motion table using the positions of the light source and the user as indicators, and the computer graphics are generated by stopping the operation of light generation units that are non-operating based on the acquired motion / non-motion information. This allows the operation of light generation units that have little impact on the user to be stopped, minimizing the degradation in the quality of the computer graphics presented to the user and reducing the amount of calculation required to generate the computer graphics.

[0422] Other Embodiments Although the embodiments have been described above, the present disclosure is not limited to the above-described embodiments.

[0423] For example, the sound reproduction system described in the above embodiment may be realized as a single device including all of the components, or may be realized by allocating each function to multiple devices and coordinating these multiple devices. In the latter case, the device corresponding to the information processing device may be an information processing device such as a smartphone, tablet terminal, or PC. For example, in the sound reproduction system 100 having a function as a renderer that generates an audio signal with added sound effects, all or part of the renderer's functions may be performed by a server. That is, all or part of the acquisition unit 111, path calculation unit 121, output sound generation unit 131, and signal output unit 141 may be located on a server (not shown). In this case, the sound reproduction system 100 is realized by combining, for example, an information processing device such as a computer or smartphone, a sound presentation device such as a head-mounted display (HMD) or earphones worn by the user 99, and a server (not shown). Note that the computer, sound presentation device, and server may be connected to each other so as to be able to communicate with each other via the same network, or may be connected via different networks. When the sound reproduction system 100 is connected via different networks, the possibility of communication delays increases, so processing on the server may be permitted only when the computer, sound presentation device, and server are connected so as to be able to communicate with each other via the same network. Also, depending on the amount of bitstream data received by the sound reproduction system 100, it may be determined whether the server will take on all or part of the functions of the renderer.

[0424] The sound reproduction system of the present disclosure can also be realized as an information processing device that is connected to a reproduction device having only a driver and that simply reproduces an output sound signal generated based on acquired sound information to the reproduction device. In this case, the information processing device may be realized as hardware having a dedicated circuit, or as software that causes a general-purpose processor to execute specific processing.

[0425] In the above-described embodiment, the processing performed by a specific processing unit may be performed by another processing unit. The order of multiple processing operations may be changed, or multiple processing operations may be performed in parallel.

[0426] In the above-described embodiments, each component may be realized by executing a software program suitable for that component, or by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0427] Furthermore, each component may be realized by hardware. For example, each component may be a circuit (or integrated circuit). These circuits may form a single circuit as a whole, or each may be a separate circuit. Furthermore, each of these circuits may be a general-purpose circuit or a dedicated circuit.

[0428] Furthermore, the general or specific aspects of the present disclosure may be realized as an apparatus, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, etc. Furthermore, the general or specific aspects of the present disclosure may be realized as any combination of an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0429] For example, the present disclosure may be realized as an audio signal reproducing method executed by a computer, or as a program for causing a computer to execute the audio signal reproducing method. The present disclosure may also be realized as a computer-readable non-transitory recording medium on which such a program is recorded.

[0430] In addition, this disclosure also includes forms obtained by applying various modifications to each embodiment that a person skilled in the art would think of, or forms realized by arbitrarily combining the components and functions of each embodiment within the scope that does not deviate from the intent of this disclosure.

[0431] In the present disclosure, the encoded sound information can be rephrased as a bitstream containing a sound signal, which is information about a predetermined sound to be reproduced by the sound reproduction system 100, and metadata, which is information about the localization position when the sound image of the predetermined sound is localized at a predetermined position within a three-dimensional sound field. For example, the sound information may be acquired by the sound reproduction system 100 as a bitstream encoded in a predetermined format such as MPEG-H 3D Audio (ISO / IEC 23008-3). As an example, the encoded sound signal includes information about the predetermined sound to be reproduced by the sound reproduction system 100. The predetermined sound here refers to a sound emitted by a sound source object present in the three-dimensional sound field or a natural environmental sound, and may include, for example, a mechanical sound or the sounds of animals, including humans. When multiple sound source objects are present in the three-dimensional sound field, the sound reproduction system 100 acquires multiple sound signals corresponding to the multiple sound source objects.

[0432] On the other hand, metadata is, for example, information used to control acoustic processing of a sound signal in the sound reproduction system 100. Metadata may be information used to describe a scene expressed in a virtual space (three-dimensional sound field). Here, a scene is a term that refers to a collection of all elements representing three-dimensional images and acoustic events in a virtual space, modeled by the sound reproduction system 100 using metadata. In other words, the metadata here may include not only information for controlling acoustic processing, but also information for controlling video processing. Of course, the metadata may include information for controlling only either audio processing or video processing, or may include information used to control both. In the present disclosure, the bitstream acquired by the sound reproduction system 100 may include such metadata. Alternatively, the sound reproduction system 100 may acquire the metadata separately from the bitstream, as described below.

[0433] The sound reproduction system 100 generates virtual sound effects by performing sound processing on the sound signal using metadata included in the bitstream and additionally acquired position information of the interactive user 99. For example, sound effects such as early reflection sound generation, late reverberation sound generation, diffraction sound generation, distance attenuation effect, localization, sound image localization processing, and Doppler effect may be added. Information for switching on and off all or part of the sound effects may also be added as metadata.

[0434] All or part of the metadata may be obtained from sources other than the bitstream of audio information. For example, either the metadata controlling audio or the metadata controlling video may be obtained from sources other than the bitstream, or both may be obtained from sources other than the bitstream.

[0435] Furthermore, if metadata for controlling the video is included in the bitstream acquired by the audio reproduction system 100, the audio reproduction system 100 may have a function for outputting the metadata that can be used for controlling the video to a display device that displays images or a 3D video reproduction device that reproduces 3D video.

[0436] As an example, the encoded metadata includes information about a three-dimensional sound field including a sound source object emitting a sound and an obstacle object, and information about a localization position when a sound image of the sound is localized at a predetermined position in the three-dimensional sound field (i.e., the sound is perceived as arriving from a predetermined direction), i.e., information about the predetermined direction. Here, an obstacle object is an object that can affect the sound perceived by the user 99 by, for example, blocking or reflecting the sound emitted by the sound source object before it reaches the user 99. Obstacle objects may include not only stationary objects but also animals such as people or moving objects such as machines. Furthermore, when multiple sound source objects exist in a three-dimensional sound field, the other sound source objects may be obstacle objects for any one sound source object. Furthermore, both non-sound-source objects such as building materials or inanimate objects and sound-emitting sound objects may be obstacle objects.

[0437] The spatial information constituting the metadata may include not only the shape of the three-dimensional sound field, but also information representing the shape and position of obstacle objects present in the three-dimensional sound field and the shape and position of sound source objects present in the three-dimensional sound field. The three-dimensional sound field may be either a closed space or an open space, and the metadata may include information representing the reflectance of structures that can reflect sound in the three-dimensional sound field, such as floors, walls, or ceilings, and the reflectance of obstacle objects present in the three-dimensional sound field. Here, the reflectance is the ratio of the energy of reflected sound to incident sound, and is set for each frequency band of sound. Of course, the reflectance may be set uniformly regardless of the frequency band of the sound. Furthermore, when the three-dimensional sound field is an open space, parameters such as a uniform attenuation rate, diffracted sound, or early reflection sound may be used.

[0438] In the above description, reflectance is cited as a parameter related to an obstacle object or a sound source object included in the metadata, but the metadata may include information other than reflectance. For example, information about the material of the object may be included as metadata related to both the sound source object and the non-sound source object. Specifically, the metadata may include parameters such as diffusion rate, transmittance, or sound absorption rate.

[0439] Information about the sound source object may include information such as volume, radiation characteristics (directivity), playback conditions, the number and type of sound sources emitted from a single object, or information specifying a sound source area within the object. The playback conditions may, for example, determine whether the sound is a continuous sound or an event-triggering sound. The sound source area within the object may be determined based on the relative relationship between the position of the user 99 and the position of the object, or may be determined based on the object. When the sound source area is determined based on the relative relationship between the position of the user 99 and the position of the object, the surface from which the user 99 is viewing the object is used as the reference, and the user 99 can be made to perceive sound X as emanating from the right side of the object and sound Y as emanating from the left side of the object as viewed from the user 99. When the sound source area is determined based on the object, the surface from which the user 99 is viewing the object is used as the reference, and the sound emitted from which area of ​​the object can be fixed regardless of the direction the user 99 is viewing. For example, the user 99 can be made to perceive a high-pitched sound coming from the right side and a low-pitched sound coming from the left side when viewing the object from the front. In this case, when the user 99 goes around to the back of the object, the user 99 can be made to perceive a low-pitched sound coming from the right side and a high-pitched sound coming from the left side as viewed from the back.

[0440] The spatial metadata may include the time to early reflections, the reverberation time, or the ratio of direct sound to diffuse sound. If the ratio of direct sound to diffuse sound is zero, the user 99 will perceive only direct sound.

[0441] Furthermore, information indicating the position and orientation of the user 99 in the three-dimensional sound field may be included in the bitstream as metadata as an initial setting, or may not be included in the bitstream. If the information indicating the position and orientation of the user 99 is not included in the bitstream, the information indicating the position and orientation of the user 99 is acquired from information other than the bitstream. For example, the position information of the user 99 in the VR space may be acquired from an app that provides VR content. The position information of the user 99 for presenting sound as AR may be position information obtained by a mobile terminal performing self-position estimation using GPS, a camera, LiDAR (Laser Imaging Detection and Ranging), or the like. Note that the sound signal and metadata may be stored in a single bitstream or may be stored separately in multiple bitstreams. Similarly, the sound signal and metadata may be stored in a single file or may be stored separately in multiple files.

[0442] When an audio signal and metadata are stored separately in multiple bitstreams, information indicating other related bitstreams may be included in one or some of the multiple bitstreams in which the audio signal and metadata are stored. Also, information indicating other related bitstreams may be included in metadata or control information of each of the multiple bitstreams in which the audio signal and metadata are stored. When an audio signal and metadata are stored separately in multiple files, information indicating other related bitstreams or files may be included in one or some of the multiple files in which the audio signal and metadata are stored. Also, information indicating other related bitstreams or files may be included in metadata or control information of each of the multiple bitstreams in which the audio signal and metadata are stored.

[0443] Here, the related bitstreams or files are, for example, bitstreams or files that may be used simultaneously during audio processing. Furthermore, information indicating other related bitstreams may be collectively described in the metadata or control information of one bitstream among multiple bitstreams storing audio signals and metadata, or may be separately described in the metadata or control information of two or more bitstreams among the multiple bitstreams storing audio signals and metadata. Similarly, information indicating other related bitstreams or files may be collectively described in the metadata or control information of one file among multiple files storing audio signals and metadata, or may be separately described in the metadata or control information of two or more files among the multiple files storing audio signals and metadata. Furthermore, a control file collectively describing information indicating other related bitstreams or files may be generated separately from the multiple files storing audio signals and metadata. In this case, the control file does not need to store the audio signal and metadata.

[0444] Here, the information indicating the other related bitstream or file may be, for example, an identifier indicating the other bitstream, a file name indicating the other file, a URL (Uniform Resource Locator), or a URI (Uniform Resource Identifier). In this case, the acquisition unit identifies or acquires the bitstream or file based on the information indicating the other related bitstream or file. Furthermore, the information indicating the other related bitstream may be included in metadata or control information of at least some of the bitstreams among a plurality of bitstreams storing audio signals and metadata, and the information indicating the other related file may be included in metadata or control information of at least some of the files among a plurality of files storing audio signals and metadata. Here, the file including information indicating the related bitstream or file may be, for example, a control file such as a manifest file used for content distribution.

[0445] The present disclosure is useful in reproducing sound, for example, by allowing a user to perceive stereoscopic sound.

[0446] 99 User 100 Sound reproduction system 101 Information processing device 102 Communication module 103 Detector 104 Driver 105 Database 111 Acquisition unit 112 Encoded sound information input unit 113 Decode processing unit 114 Sensing information input unit 115 Table setting unit 116 Table reference unit 121 Path calculation unit 131 Output sound generation unit 132 Reduction processing unit 133 Culling unit 134 Integration unit 141 Signal output unit 300 3D video reproduction device

Claims

1. A sound processing device comprising: an acquisition unit that acquires sound information including an acoustic signal and information on the position of a sound source object in a three-dimensional sound field; a table reference unit that references a table that associates at least one of the position of a user and the position of the sound source object in the three-dimensional sound field with information for determining a target sound; and a reduction processing unit that determines the target sound using the information for determining the target sound obtained by referencing the table, and generates an output sound signal that does not include the signal by reducing the signal of the determined target sound from multiple sound signals generated when generating an output sound signal from the acoustic signal included in the acquired sound information.

2. The sound processing device according to claim 1, wherein the table includes a priority order for each of the plurality of sounds, the lower the priority order, the more likely the sound is to be determined as the target sound.

3. The sound processing device according to claim 1, wherein the table includes an identifier for distinguishing each of at least two of the plurality of sounds determined to be the target sound from other sounds.

4. The sound processing device according to claim 1, wherein the table includes information indicating whether or not to perform an operation to generate each of the plurality of sounds.

5. The sound processing device of claim 1, wherein the table includes information indicating whether or not to perform an operation to generate each of the plurality of sounds, and a priority order for each of the plurality of sounds, the lower the priority order, the more likely the sound is to be determined as the target sound.

6. The sound processing device according to claim 1, wherein the reduction processing unit includes a culling unit that reduces the target sound signal by discarding the target sound signal.

7. The sound processing device according to claim 6, wherein the reduction processing unit discards the target sound signal by stopping an operation of generating the target sound signal.

8. The sound processing device according to claim 1, wherein the reduction processing unit includes an integration unit that reduces the signals of the at least two sounds of the targets by discarding the signals of the at least two sounds of the targets and supplementing the signals of the at least two sounds of the targets with a single virtual sound signal obtained by integrating the signals of the at least two sounds of the targets.

9. The sound processing device according to claim 8, wherein the integration unit supplements the signal of the virtual sound so that the virtual sound is localized at a position based on the positions of the at least two discarded target sounds and the position of the user.

10. The sound processing device described in claim 3, wherein the table is created in advance to include identifiers of at least two sounds determined to be at least two of the target sounds based on at least one of information regarding the angle of two or more sounds arriving toward the user and information regarding the level ratio of two or more sounds arriving toward the user.

11. The sound processing device according to claim 1, wherein the reduction processing unit performs the reduction of at least one sound signal gradually in the time domain.

12. The sound processing device according to claim 1, wherein the reduction processing unit corrects information for determining the target sound obtained by referring to the table, and determines the target sound using the corrected information for determining the target sound.

13. The sound processing device according to claim 1, wherein the table reference unit converts each of the user's position and the sound source object's position using the positions of lattice points that divide the three-dimensional sound field into unit spaces of a predetermined size, and references the table using at least one of the converted user's position and sound source object's position.

14. The sound processing device according to claim 1, wherein each of the user's position and the sound source object's position is converted by an identifier of a voxel, which is a unit space that divides the three-dimensional sound field into a plurality of sizes, and the table is referenced using at least one of the converted user's position and the sound source object's position.

15. The sound processing device according to claim 13 or 14, wherein the size of the unit space is changed based on at least one of a memory capacity and a processing capacity of the sound processing device.

16. The sound processing device according to claim 1, further comprising a table setting unit that sets the table when the sound processing device is initialized.

17. The sound processing device according to claim 16, wherein the table setting section further updates at least a part of the table when information for updating the table is acquired.

18. The sound processing device according to claim 2, wherein the reduction processing unit accumulates the amount of calculation in descending order of the priority included in the table, and determines as the target sound a sound having a rank lower than or equal to the amount of calculation that exceeds a predetermined amount of calculation.

19. The sound processing device according to claim 1, wherein the table reference unit, when there are multiple users or multiple sound source objects, references one of the multiple tables corresponding to a combination of the position of the user and the position of the sound source object.

20. An acoustic processing method executed by a computer, comprising: a step of acquiring sound information including an acoustic signal and information on the position of a sound source object in a three-dimensional sound field; a step of referring to a table that corresponds at least one of the position of a user and the position of the sound source object in the three-dimensional sound field to information for determining a target sound; and a step of determining the target sound using the information for determining the target sound obtained by referring to the table, and generating an output sound signal that does not include the determined target sound signal by reducing the signal of the determined target sound from multiple sound signals generated when generating an output sound signal from the acoustic signal included in the acquired sound information.

21. A program for causing a computer to execute the acoustic processing method according to claim 20.

Images