Acoustic processing device, acoustic processing method, and program

Abstract

An acoustic processing device (information processing device (101)) comprises: an acquisition unit (111) that acquires sound information including acoustic signals and information about the state of a sound source object in a three-dimensional sound field; a correlation calculation unit (115) that calculates a first correlation, which is the correlation between the sound source object and the state of a user (99) in the three-dimensional sound field at a prescribed point in time, and a second correlation, which is the correlation between the sound source object and the state of the user in the three-dimensional sound field at a subsequent point in time after the prescribed point in time; and a reduction processing unit (132) that, in accordance with the first correlation and the second correlation, reduces a sound signal from among a plurality of sound signals generated when output sound signals are generated from the acoustic signals included in the acquired sound information, and thereby generates an output sound signal not included in the aforementioned sound signal.

Description

Sound processing device, sound processing method, and program

[0001] The present disclosure relates to a sound processing device, a sound processing method, and a program.

[0002] Conventionally, there are known techniques for reproducing sound in a virtual three-dimensional space to allow a user to perceive stereoscopic sound (see, for example, Patent Document 1). Furthermore, in order to perceive sound as if it is coming from a sound source object to the user in such a three-dimensional space, processing is required to generate output sound information from original sound information. In particular, reproducing stereoscopic sound in response to the user's body movements in a virtual space requires extensive processing. Advances in computer graphics (CG) have made it relatively easy to create visually complex virtual environments, making technology for realizing corresponding auditory information important. Additionally, when processing from sound information to output sound information is performed in advance, a large memory area is required to store the pre-calculated processing results. Furthermore, transmitting such large amounts of processed data may require a wide communication bandwidth.

[0003] To achieve a more realistic sound environment, the number of objects that emit sound in the virtual three-dimensional space must be increased, and secondary sounds based on acoustic effects such as reflected sound, diffracted sound, and reverberation must be increased. Furthermore, these secondary sounds must be appropriately changed in response to the user's movements, which requires a large amount of processing power.

[0004] Japanese Patent Application Laid-Open No. 2020-18620

[0005] Therefore, an object of the present disclosure is to provide a sound processing device or the like that can generate an output sound signal appropriately in terms of the amount of processing.

[0006] A sound processing device according to one aspect of the present disclosure includes an acquisition unit that acquires sound information including an acoustic signal and information about the state of a sound source object in a three-dimensional sound field; a relative relationship calculation unit that calculates a first relative relationship, which is the relative relationship between the state of the sound source object and a user in the three-dimensional sound field at a predetermined time point, and a second relative relationship, which is the relative relationship between the state of the sound source object and the user in the three-dimensional sound field at a subsequent time point after the predetermined time point; and a reduction processing unit that generates an output sound signal that does not include a plurality of sound signals generated when generating an output sound signal from the acoustic signal included in the acquired sound information by reducing sound signals according to the first relative relationship and the second relative relationship.

[0007] Furthermore, an acoustic processing method according to one aspect of the present disclosure is an acoustic processing method executed by a computer, and includes the steps of: acquiring sound information including an acoustic signal and information about the state of a sound source object in a three-dimensional sound field; calculating a first relative relationship which is the relative relationship of the state between the sound source object and a user in the three-dimensional sound field at a predetermined time point; and calculating a second relative relationship which is the relative relationship of the state between the sound source object and the user in the three-dimensional sound field at a subsequent time point after the predetermined time point; and generating an output sound signal that does not include the signal by reducing sound signals from among a plurality of sound signals generated when generating an output sound signal from the acoustic signal included in the acquired sound information, according to the first relative relationship and the second relative relationship.

[0008] Furthermore, one aspect of the present disclosure can also be realized as a program for causing a computer to execute the above-described acoustic processing method.

[0009] These comprehensive or specific aspects may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0010] According to the present disclosure, it is possible to appropriately generate an output sound signal.

[0011] FIG. 1 is a schematic diagram showing a use example of a sound reproduction system according to an embodiment. FIG. 2 is a block diagram showing a functional configuration of the sound reproduction system according to an embodiment. FIG. 3 is a diagram for explaining an example of an audio signal according to an embodiment. FIG. 4 is a block diagram showing a functional configuration of an acquisition unit according to an embodiment. FIG. 5 is a block diagram showing a functional configuration of an output sound generation unit according to an embodiment. FIG. 6 is a diagram for explaining another example of a sound reproduction system according to an embodiment. FIG. 7 is a diagram for explaining another example of a sound reproduction system according to an embodiment. FIG. 8 is a diagram for explaining another example of a sound reproduction system according to an embodiment. FIG. 9 is a diagram for explaining another example of a sound reproduction system according to an embodiment. FIG. 10 is a diagram for explaining another example of a sound reproduction system according to an embodiment. FIG. 11 is a diagram for explaining another example of a sound reproduction system according to an embodiment. FIG. 12 is a diagram for explaining another example of a sound reproduction system according to an embodiment. FIG. 13 is a diagram for explaining another example of a sound reproduction system according to an embodiment. FIG. 14 is a diagram for explaining another example of a sound reproduction system according to an embodiment. FIG. 15 is a diagram for explaining a specific example of a sound reproduction system according to an example of an embodiment. FIG. 16 is a diagram for explaining a specific example of a sound reproduction system according to an example of an embodiment. FIG. 17 is a diagram for explaining a specific example of a sound reproduction system according to an example of an embodiment. FIG. 18 is a diagram for explaining a specific example of a sound reproduction system according to an example of an embodiment. FIG. 19 is a diagram for explaining a specific example of a sound reproduction system according to an example of an embodiment. FIG. 20 is a diagram for explaining a specific example of a sound reproduction system according to an example of an embodiment. FIG. 21 is a diagram for explaining a specific example of a sound reproduction system according to an example of an embodiment. FIG. 22 is a diagram for explaining a specific example of a sound reproduction system according to an example of an embodiment. FIG. 23 is a diagram for explaining a specific example of a sound reproduction system according to an example of an embodiment. FIG. 24 is a diagram for explaining a specific example of a sound reproduction system according to an example of an embodiment. FIG. 25 is a diagram for explaining a specific example of a sound reproduction system according to an example of an embodiment.FIG. 26 is a diagram for explaining a specific example of a sound reproduction system according to an example of an embodiment. FIG. 27 is a diagram for explaining a specific example of a sound reproduction system according to an example of an embodiment. FIG. 28 is a diagram for explaining a specific example of a sound reproduction system according to an example of an embodiment. FIG. 29 is a diagram for explaining a specific example of a sound reproduction system according to an example of an embodiment. FIG. 30 is a diagram for explaining a specific example of a sound reproduction system according to an example of an embodiment. FIG. 31 is a diagram for explaining a specific example of a sound reproduction system according to an example of an embodiment. FIG. 32 is a diagram for explaining a specific example of a sound reproduction system according to an example of an embodiment. FIG. 33 is a diagram for explaining a specific example of a sound reproduction system according to an example of an embodiment.

[0012] (Knowledge that forms the basis of the disclosure) Conventionally, a technology related to sound reproduction that allows a user to perceive stereoscopic sound in a virtual three-dimensional space (hereinafter sometimes referred to as a three-dimensional sound field) has been known (see, for example, Patent Document 1). Using this technology, a user can perceive sound as if a sound source object exists at a predetermined position in the virtual space and the sound is coming from that direction. In order to localize a sound image at a predetermined position in the virtual three-dimensional space in this way, for example, calculation processing is required for a sound signal emitted by a sound source object (also referred to as a sound emitted from the sound source object or a reproduced sound) to generate a sound arrival time difference between the two ears and a sound level difference (or sound pressure difference) between the two ears that causes the sound to be perceived as stereoscopic sound. Such calculation processing is performed by applying a stereophonic filter. A stereophonic filter is an information processing filter that, when an output sound signal obtained by applying the filter to original sound information is reproduced, causes the position (such as the direction and distance) of the sound, the size of the sound source, the width of the space, and the like to be perceived with a three-dimensional effect.

[0013] As an example of the computational process for applying such a stereophonic filter, a process is known in which a head-related transfer function (HRTF) is convolved with a target sound signal to make the sound perceived as coming from a predetermined direction. By performing this HRTF convolution process at a sufficiently fine angle with respect to the direction of arrival of the reproduced sound from the position of the sound source object to the user's position, the sense of realism experienced by the user is improved.

[0014] In recent years, there has been active development of technologies related to virtual reality (VR). Virtual reality focuses on appropriately changing the position of a sound source object in a virtual three-dimensional space in response to a user's movements, allowing the user to experience the sensation of moving within the virtual space. To achieve this, it is necessary to move the localization position of a sound image in the virtual space relative to the user's movements. This processing has been performed by applying a stereophonic filter, such as the head-related transfer function described above, to the original sound information. However, when a user moves within a three-dimensional space, the sound transmission path changes from moment to moment due to changes in the positional relationship between the sound source object and the user, such as due to sound reverberation and interference. This requires determining the sound transmission path from the sound source object based on the positional relationship between the sound source object and the user, and convolving the transfer function to take into account sound reverberation and interference. However, such information processing requires a huge amount of processing, and an improvement in the sense of realism may not be achieved without a large-scale processing device.

[0015] Therefore, in order to reduce such an enormous amount of processing, attempts have been made to reduce a portion of the sounds being reproduced. Specifically, for each of the multiple sound source objects in a three-dimensional space, or for each of the multiple types of sounds generated from each of the sound source objects, rather than convolving a head-related transfer function with each of them, a portion of the sounds is reduced and then the head-related transfer function is convolved. By doing so, in the convolution of the head-related transfer function, which requires a particularly large amount of processing, that is, in the process of generating a stereophonic signal for output (in other words, an output signal or an output sound signal), the number of sound signals to be processed is reduced, and a significant reduction in the processing amount is expected.

[0016] However, since indiscriminate reduction of sound signals will result in sound degradation, it is necessary to reduce sounds that cause relatively little sound degradation in order to suppress this sound degradation. Here, let's assume that sounds that are suitable for reduction at a subsequent point in time are reduced in advance at a certain point in time. Although the sounds reduced at the subsequent point in time will also be reduced at the previous point in time, the user will be less likely to feel discomfort due to the sound reduction because the reduced sounds are continuous. In other words, while minimizing the discomfort felt by the user, sound reduction can be started a little earlier, making it easier to increase the number of sound signals to be reduced, thereby improving the effectiveness of reducing the amount of processing. This makes it possible to realize a sound processing device that can suppress sound degradation while reducing the amount of processing.

[0017] A more specific outline of the present disclosure is as follows.

[0018] A sound processing device according to a first aspect of the present disclosure includes an acquisition unit that acquires sound information including an acoustic signal and information on the state of a sound source object in a three-dimensional sound field; a relative relationship calculation unit that calculates a first relative relationship, which is the relative relationship between the state of the sound source object and a user in the three-dimensional sound field at a predetermined time point, and a second relative relationship, which is the relative relationship between the state of the sound source object and the user in the three-dimensional sound field at a subsequent time point after the predetermined time point; and a reduction processing unit that generates an output sound signal that does not include a plurality of sound signals generated when generating an output sound signal from the acoustic signal included in the acquired sound information by reducing sound signals according to the first relative relationship and the second relative relationship.

[0019] In other words, the sound processing device of the first aspect is a sound processing device that predicts the relative relationship between the states of the listener (user) and the object (sound source object) Δt later based on the relative relationship between the states of the listener (user) and the object (sound source object) at time t, and applies a predetermined processing (reduction processing) to the sound that reaches the listener in the state at time t and the sound that reaches the listener in the state at time t + Δt.

[0020] According to such a sound processing device, by reducing sounds suitable for reduction at a subsequent time point following a predetermined time point in advance at the predetermined time point, it is possible to minimize the sense of discomfort felt by the user while starting sound reduction from a slightly earlier time point, making it easier to increase the number of sound signals to be reduced and improving the effect of reducing the amount of processing. Thus, it is possible to generate an output sound signal appropriately in terms of the amount of processing.

[0021] A sound processing device according to a second aspect is the sound processing device according to the first aspect, wherein the state is a position in a three-dimensional sound field.

[0022] That is, the sound processing device according to the second aspect is a sound processing device in which the state is a spatial position.

[0023] According to such a sound processing device, when calculating the first relative relationship and the second relative relationship, the spatial position can be used as the state.

[0024] A sound processing device according to a third aspect is the sound processing device according to the first aspect, wherein the state is an orientation in a three-dimensional sound field determined by at least one of an azimuth angle and an elevation angle.

[0025] That is, the sound processing device according to the third aspect is a sound processing device whose state is an orientation determined by at least one of an azimuth angle and an elevation angle.

[0026] According to such a sound processing device, when calculating the first relative relationship and the second relative relationship, the direction in the three-dimensional sound field determined by at least one of the azimuth angle and the elevation angle can be used as the state.

[0027] Furthermore, a sound processing device according to a fourth aspect is the sound processing device according to the first aspect, wherein the state is a position within a three-dimensional sound field and an orientation within the three-dimensional sound field determined by at least one of an azimuth angle or an elevation angle, and the time interval between a predetermined point in time and a subsequent point in time is different between the first relative relationship and the second relative relationship when the state is a position within the three-dimensional sound field and the first relative relationship and the second relative relationship when the state is an orientation within the three-dimensional sound field.

[0028] That is, the sound processing device according to the fourth aspect is a sound processing device in which the state is a position and an orientation, and the prediction of the position and the orientation is performed at different time resolutions.

[0029] According to such a sound processing device, when calculating the first relative relationship and the second relative relationship, a position in the three-dimensional sound field and an orientation in the three-dimensional sound field determined by at least one of an azimuth angle and an elevation angle can be used as states, and the time interval between a predetermined time point and a subsequent time point for a position in the three-dimensional sound field can be made different from the time interval between a predetermined time point and a subsequent time point for an orientation in the three-dimensional sound field.

[0030] Furthermore, a sound processing device according to a fifth aspect is the sound processing device according to the first aspect, wherein the state is at least one of a position within a three-dimensional sound field and an orientation within the three-dimensional sound field determined by at least one of an azimuth angle or an elevation angle, and the reduction processing unit reduces the sound signal depending on whether the change over time in the relative relationship indicated by the second relative relationship with respect to the first relative relationship indicates movement in a direction away from the sound source object and the user.

[0031] That is, the sound processing device according to the fifth aspect is a sound processing device in which the state is at least one of a spatial position and an orientation, and the relative relationship of the states is whether or not the listener and the object change in a direction away from each other.

[0032] According to such a sound processing device, when calculating the first relative relationship and the second relative relationship, at least one of the position in the three-dimensional sound field and the orientation in the three-dimensional sound field determined by at least one of the azimuth angle and the elevation angle can be used as a state, and sound signals can be reduced depending on whether or not the state indicates movement in a direction away from the sound source object and the user.

[0033] Furthermore, a sound processing device according to a sixth aspect is a sound processing device according to any one of the first to fifth aspects, in which the reduction processing unit reduces sound signals from among a plurality of sound signals in accordance with the importance set for each sound to the user and the first relative relationship and the second relative relationship.

[0034] That is, the sound processing device according to the sixth aspect is a sound processing device in which the predetermined processing selects sounds that are less likely to be processed according to the importance to the listener.

[0035] According to such a sound processing device, it is possible to reduce sound signals in accordance with the importance set for each sound to the user and the first and second relative relationships.

[0036] Furthermore, a sound processing device according to a seventh aspect is the sound processing device according to the sixth aspect, wherein the reduction processing unit reduces sound signals among the plurality of sound signals in accordance with the importance set for each sound to the user and the first relative relationship and the second relative relationship when the difference between the first relative relationship and the second relative relationship exceeds a predetermined threshold value.

[0037] In other words, the sound processing device of the seventh aspect is a sound processing device that performs processing according to the importance of the sound reaching the listener when a predetermined process detects a sudden change in the relative relationship between the state of the listener and the state of the object after Δt.

[0038] According to such an audio processing device, when the difference between the first relative relationship and the second relative relationship exceeds a predetermined threshold, sound signals from among multiple sound signals can be reduced according to the importance set for each sound to the user and the first relative relationship and the second relative relationship.

[0039] Furthermore, an audio processing device according to an eighth aspect is an audio processing device according to any one of the first to fifth aspects, in which the reduction processing unit includes a culling unit that reduces the sound signal by discarding the sound signal.

[0040] That is, the sound processing device according to the eighth aspect is a sound processing device in which the predetermined processing is to cull sounds with low importance.

[0041] According to such a sound processing device, the sound signal can be reduced by discarding the sound signal.

[0042] Furthermore, a sound processing device according to a ninth aspect is the sound processing device according to the eighth aspect, wherein the reduction processing unit includes an integration unit that reduces the at least two sound signals by discarding at least two sound signals and supplementing the at least two sound signals with one virtual sound signal that is an integration of the at least two sound signals.

[0043] That is, the sound processing device according to the ninth aspect is a sound processing device in which the predetermined process outputs sounds represented by a number of virtual objects that is smaller than the number of sounds with low importance.

[0044] According to such a sound processing device, it is possible to reduce the at least two sound signals by discarding the at least two sound signals and supplementing them with a single virtual sound signal that is an integration of the at least two sound signals.

[0045] Furthermore, a sound processing device according to a tenth aspect is the sound processing device according to the third aspect, in which the reduction processing unit prohibits reduction of the sound signal according to the second relative relationship when a change in the state of at least one of the sound source object and the user between a predetermined point in time and a subsequent point in time satisfies a predetermined condition.

[0046] In other words, the sound processing device of the tenth aspect is a sound processing device that stops predetermined processing for sound that reaches the listener in a state at time t + Δt when a change in the state of at least one of the listener or the object satisfies a predetermined condition.

[0047] According to such a sound processing device, when the change in the state of at least one of the sound source object and the user between a predetermined point in time and a subsequent point in time satisfies a predetermined condition, reduction of the sound signal according to the second relative relationship can be prohibited.

[0048] Furthermore, an audio processing device according to an eleventh aspect is the audio processing device according to the tenth aspect, wherein the predetermined condition is that the change in the state of at least one of the sound source object and the user between a predetermined point in time and a subsequent point in time is discontinuous.

[0049] That is, the sound processing device according to the eleventh aspect is a sound processing device in which the predetermined condition is that the state changes discontinuously.

[0050] According to such a sound processing device, when the change in the state of at least one of the sound source object and the user between a predetermined point in time and a subsequent point in time is discontinuous, reduction of the sound signal according to the second relative relationship can be prohibited.

[0051] Furthermore, a sound processing device according to a twelfth aspect is a sound processing device according to the tenth or eleventh aspect, in which the predetermined condition is that the rate of change in the state of at least one of the sound source object and the user between a predetermined point in time and a subsequent point in time exceeds a predetermined rate.

[0052] That is, the sound processing device according to the twelfth aspect is a sound processing device in which the predetermined condition is that the state changes faster than a predetermined speed.

[0053] According to such a sound processing device, when the rate of change in the state of at least one of the sound source object and the user between a predetermined point in time and a subsequent point in time exceeds a predetermined rate, reduction of the sound signal according to the second relative relationship can be prohibited.

[0054] Furthermore, a sound processing device according to a thirteenth aspect is a sound processing device according to any one of the tenth to twelfth aspects, wherein the specified condition indicates the occurrence of an obstacle between the sound source object and the user.

[0055] That is, the sound processing device according to the thirteenth aspect is a sound processing device in which the predetermined condition is that an obstacle is detected between the listener and the object.

[0056] According to such a sound processing device, when an obstacle is detected between the sound source object and the user, it is possible to prohibit reduction of the sound signal according to the second correlation.

[0057] Furthermore, a sound processing device according to a fourteenth aspect is a sound processing device according to any one of the tenth to thirteenth aspects, in which the predetermined condition is that the deviation between the calculated state of at least one of the sound source object and the user at a subsequent point in time and the actual measured value exceeds a predetermined threshold value.

[0058] That is, the sound processing device according to the fourteenth aspect is a sound processing device in which the predetermined condition is that the difference between the predicted value of the state at time t+Δt and the actual state exceeds a predetermined threshold.

[0059] According to such an audio processing device, if the deviation between the calculated result of the state of at least one of the sound source object and the user at a subsequent point in time and the actual measured value exceeds a predetermined threshold, reduction of the sound signal according to the second relative relationship can be prohibited.

[0060] Furthermore, an acoustic processing method according to a fifteenth aspect is an acoustic processing method executed by a computer, and includes the steps of acquiring sound information including an acoustic signal and information about the state of a sound source object in a three-dimensional sound field, calculating a first relative relationship which is the relative relationship between the state of the sound source object and the user in the three-dimensional sound field at a predetermined time point, and a second relative relationship which is the relative relationship between the state of the sound source object and the user in the three-dimensional sound field at a subsequent time point after the predetermined time point, and generating an output sound signal that does not include the signal by reducing sound signals from among a plurality of sound signals generated when generating an output sound signal from the acoustic signal included in the acquired sound information, in accordance with the first relative relationship and the second relative relationship.

[0061] This can provide the same effects as the sound processing device described above.

[0062] A program according to a sixteenth aspect is a program for causing a computer to execute the above-described acoustic processing method.

[0063] This makes it possible to achieve the same effects as the above-described acoustic processing method using a computer.

[0064] Furthermore, these comprehensive or specific aspects may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0065] Hereinafter, embodiments will be described in detail with reference to the drawings. Note that the embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not recited in independent claims will be described as optional components. Note that each figure is a schematic diagram and is not necessarily an exact illustration. Furthermore, in each figure, substantially identical components are assigned the same reference numerals, and duplicated descriptions may be omitted or simplified.

[0066] In the following description, elements may be assigned ordinal numbers such as first, second, and third. These ordinal numbers are assigned to elements in order to identify them and do not necessarily correspond to a meaningful order. These ordinal numbers may be rearranged, newly added, or removed as appropriate.

[0067] In the following description, an acoustic signal included in sound information may be described, but the acoustic signal may also be referred to as a voice signal or a sound signal. In other words, in the present disclosure, the acoustic signal has the same meaning as the voice signal or the sound signal.

[0068] (Embodiment) [Overview] First, an overview of an audio reproduction system according to an embodiment will be described. Fig. 1 is a schematic diagram showing an example of use of an audio reproduction system according to an embodiment. Fig. 1 shows a user 99 using an audio reproduction system 100.

[0069] The sound reproduction system 100 shown in FIG. 1 is used simultaneously with, for example, a three-dimensional video reproduction device 300. By simultaneously viewing three-dimensional images and three-dimensional sound, the image enhances the auditory sense of realism, and the sound enhances the visual sense of realism, allowing the user to experience the image and sound as if they were actually at the scene where they were captured. For example, when an image (moving image) of people having a conversation is displayed, it is known that even if the localization of the sound image (sound source object) of the conversation sound is not aligned with the person's mouth, the user 99 will perceive the conversation sound as coming from the person's mouth. In this way, the visual information can correct the position of the sound image, and the image and sound can be combined to enhance the sense of realism.

[0070] The three-dimensional video reproduction device 300 is an image display device worn on the head of the user 99. Therefore, the three-dimensional video reproduction device 300 moves integrally with the head of the user 99. For example, as shown in the figure, the three-dimensional video reproduction device 300 is a glasses-type device that is supported by the ears and nose of the user 99.

[0071] The three-dimensional video reproduction device 300 changes the displayed image in accordance with the movement of the user 99's head, thereby making the user 99 perceive the movement of his or her head in the three-dimensional image space. In other words, when an object in the three-dimensional image space is located in front of the user 99, if the user 99 turns to the right, the object moves to the left of the user 99, and if the user 99 turns to the left, the object moves to the right of the user 99. In this way, the three-dimensional video reproduction device 300 moves the three-dimensional image space in the opposite direction to the movement of the user 99.

[0072] The 3D video playback device 300 displays two images with a parallax difference to each of the user's 99's left and right eyes. The user 99 can perceive the three-dimensional position of an object on the image based on the parallax difference between the displayed images. Note that when the user 99 uses the audio playback system 100 with their eyes closed, for example, when using it to play healing sounds for sleep induction, the 3D video playback device 300 does not need to be used at the same time. In other words, the 3D video playback device 300 is not an essential component of the present disclosure. In addition to dedicated video display devices, the 3D video playback device 300 may also be a general-purpose mobile terminal owned by the user 99, such as a smartphone or tablet device.

[0073] Such general-purpose mobile terminals are equipped with not only a display for displaying images but also various sensors for detecting the terminal's posture and movement. Furthermore, they are also equipped with a processor for information processing, and are capable of connecting to a network to transmit and receive information to and from a server device such as a cloud server. In other words, the 3D video playback device 300 and the audio playback system 100 can be realized by combining a smartphone with general-purpose headphones or the like that do not have an information processing function.

[0074] As in this example, the head movement detection function, the video presentation function, the video information processing function for presentation, the sound presentation function, and the sound information processing function for presentation may be appropriately arranged in one or more devices to realize the 3D video reproduction device 300 and the sound reproduction system 100. If the 3D video reproduction device 300 is not required, it is sufficient to appropriately arrange the head movement detection function, the sound presentation function, and the sound information processing function for presentation in one or more devices. For example, the sound reproduction system 100 can be realized by a processing device such as a computer or smartphone having a sound information processing function for presentation, and headphones or the like having a head movement detection function and a sound presentation function.

[0075] The sound reproduction system 100 is a sound presentation device that is worn on the head of the user 99. Therefore, the sound reproduction system 100 moves integrally with the head of the user 99. For example, the sound reproduction system 100 in this embodiment is a so-called over-ear headphone type device. Note that there are no particular limitations on the form of the sound reproduction system 100, and it may be, for example, two earplug-type devices that are worn independently on the left and right ears of the user 99.

[0076] The sound reproduction system 100 changes the sound presented in accordance with the movement of the head of the user 99, thereby making the user 99 perceive as if he or she is moving his or her head within the three-dimensional sound field. For this reason, as described above, the sound reproduction system 100 moves the three-dimensional sound field in the opposite direction to the movement of the user 99.

[0077] Here, when the user 99 moves within the three-dimensional sound field, the position of the sound source object changes relative to the user 99's position within the three-dimensional sound field. As a result, each time the user 99 moves, it is necessary to perform calculation processing based on the positions of the sound source object and the user 99 to generate an output sound signal for playback. Since such processing typically requires a huge amount of processing, in the present disclosure, from the perspective of reducing the amount of processing, an output sound signal is generated and output by reducing the multiple sound signals that constitute the output sound signal used for head-related transfer function convolution. As a result, the number of sound signals used for head-related transfer function convolution is reduced, and a significant reduction in processing volume is expected. In this case, if sound signals to be reduced are determined by performing additional calculations, the processing volume increases by the amount of processing for the additional calculations. Therefore, in the present disclosure, sound signals to be reduced can be determined using as simple a calculation as possible. Therefore, in the present disclosure, sound signals to be reduced can be determined using only simple calculations by referencing a pre-calculated table. In this way, with the present disclosure, it is possible to appropriately determine the sound to be reduced with only simple additional calculations, and to make the effect of reducing the amount of processing obtained by reducing the signal of that sound more pronounced.

[0078] [Configuration] Next, the configuration of the sound reproduction system 100 according to this embodiment will be described with reference to Fig. 2. Fig. 2 is a block diagram showing the functional configuration of the sound reproduction system according to this embodiment.

[0079] As shown in FIG. 2, the sound reproduction system 100 according to this embodiment includes an information processing device 101, a communication module 102, a detector 103, a driver 104, and a database 105.

[0080] The information processing device 101 is an example of an audio processing device, and is an arithmetic device for performing various signal processing in the audio reproduction system 100. The information processing device 101 includes a processor and a memory, such as a computer, and is realized by the processor executing a program stored in the memory. Execution of this program provides functions related to each functional unit described below.

[0081] The information processing device 101 includes an acquisition unit 111, a path calculation unit 121, an output sound generation unit 131, and a signal output unit 141. Details of each functional unit included in the information processing device 101 will be described below together with details of the configuration other than the information processing device 101.

[0082] The communication module 102 is an interface device for accepting input of sound information to the sound reproduction system 100. The communication module 102 includes, for example, an antenna and a signal converter, and receives sound information from an external device via wireless communication. More specifically, the communication module 102 receives, using the antenna, a wireless signal indicating sound information converted into a format for wireless communication, and reconverts the wireless signal into sound information using the signal converter. In this way, the sound reproduction system 100 acquires sound information from the external device via wireless communication. The sound information acquired by the communication module 102 is acquired by the acquisition unit 111. In this way, the acquisition unit 111 is an example of a sound acquisition unit. The sound information is input to the information processing device 101 in the above manner. Note that communication between the sound reproduction system 100 and the external device may be performed via wired communication.

[0083] The sound information acquired by the sound reproduction system 100 is encoded in a predetermined format, such as MPEG-H 3D Audio (ISO / IEC 23008-3). As an example, the encoded sound information includes information about the sound reproduced by the sound reproduction system 100 and information about the localization position when the sound image of the sound is localized at a predetermined position in a three-dimensional sound field (i.e., perceived as sound coming from a predetermined direction). The sound information can also be interpreted as information about a sound source object. In other words, the sound information includes the position of the sound source object in the three-dimensional sound field and the sound produced by the sound source object.

[0084] As described above, sound information is obtained as input data, and includes an audio signal (acoustic signal), which is information about the reproduced sound, and other information, such as information about the position of a sound source object in a three-dimensional sound field. The other information may also include information for defining a three-dimensional sound field. Therefore, the other information may be collectively referred to as information about space (spatial information), including information about the position of a sound source object and information for defining a three-dimensional sound field. When the audio signal is viewed as the main focus, the input data can be said to be sound information in which other information (metadata) is attached to the audio signal. When the spatial information is viewed as the main focus, the input data can be said to be information in which the audio signal is attached to the spatial information. Alternatively, since the input data has both aspects, the input data can also be considered as sound spatial information.

[0085] As a specific example, the sound information includes information about a plurality of sounds including a first reproduced sound and a second reproduced sound, and the sound images of the respective sounds when reproduced are localized so that they are perceived as coming from different positions in a three-dimensional sound field. Therefore, the sound source object of the first reproduced sound is localized at a first position in the three-dimensional sound field, and the sound source object of the second reproduced sound is localized at a second position in the three-dimensional sound field. Thus, the sound information may include a plurality of sounds. In other words, the sound information may include a plurality of audio signals corresponding to the first reproduced sound and the second reproduced sound, respectively, and the positions of a plurality of sound source objects at first and second positions that correspond one-to-one to the plurality of audio signals.

[0086] 3 is a diagram illustrating an example of an audio signal according to an embodiment. For example, as shown in (a) of FIG. 3, the sound information may include an audio signal of a first direct sound arriving from a first position (from a first direction) to the position of the user 99 and an audio signal of a second direct sound arriving from a second position (from a second direction) to the position of the user 99. The sound information immediately after acquisition may include only information about the reproduced sound. In this case, information about a predetermined position may be acquired separately, and the subsequent processing may be performed once the information is collected. As described above, the sound information includes first sound information about the first reproduced sound and second sound information about the second reproduced sound. However, multiple pieces of sound information each including these may be acquired separately and played back simultaneously (i.e., treated as a single piece of sound information), thereby localizing sound images at different positions in a three-dimensional sound field and causing the reproduced sounds to arrive from different directions.

[0087] Alternatively, the sound information may include multiple audio signals and the position of a single sound source object that corresponds to the multiple audio signals in a many-to-one relationship. For example, such sound information is used in a situation where multiple sounds are reproduced from a sound source object. For example, each of the multiple audio signals corresponds to a direct sound that arrives directly from the position of the sound source object to the position of the user 99, and a secondary sound (sound generated by indirect propagation) that accompanies the direct sound and arrives via a path different from that of the direct sound.

[0088] For example, as shown in (b) of FIG. 3 , the sound information immediately after acquisition includes an audio signal related to the direct sound, and is converted into sound information including audio signals such as reverberation, primary reflection, and diffraction through a conversion process for calculating secondary sounds. This conversion process for calculating secondary sounds uses information on the spatial environment conditions of the three-dimensional sound field (e.g., the position, reflection, and diffraction characteristics of objects in the three-dimensional sound field). Thus, secondary sounds are computationally generated from sound information related to a single playback sound based on the spatial environment conditions of the three-dimensional sound field. Therefore, the secondary sounds are not included in the sound information immediately after acquisition, and sound information including these secondary sounds is generated through the conversion process for calculating the secondary sounds. Another secondary sound may be generated from one secondary sound through its propagation. Note that the information on the spatial environment conditions is part of the spatial information and may be acquired together with the audio signal from the input sound information. Alternatively, the audio signal and the spatial information may be acquired separately. That is, the sound information may be acquired from a single file or bitstream, or may be acquired separately by dividing it into multiple files or bitstreams. For example, the audio signal and the spatial information may be acquired from separate files or bitstreams, or the audio signal and the spatial information may each be acquired from a plurality of files or bitstreams.

[0089] The example in Figure 3(b) illustrates that a secondary reflected sound is generated from a primary reflected sound. As shown in the figure, these secondary sounds are assigned tags that identify their relationships with each other, such as parent, child, and grandchild, as information regarding the genealogy of their generation relationships from the direct sound (in other words, the generation system). Alternatively, when the direct sound is defined as the 0th generation, the secondary sounds may be quantified as the number of generations, such as the first generation to which the first reflected sound belongs and the second generation to which the second reflected sound belongs. Note that the number of generations that are allowed to be generated from the direct sound may be set according to the scale of the computational resources.

[0090] As described above, there is no particular limitation on the form of the input sound information, and the sound reproduction system 100 may be provided with an acquisition unit 111 that can accommodate various forms of sound information.

[0091] An example of the acquisition unit 111 will now be described with reference to Fig. 4. Fig. 4 is a block diagram showing the functional configuration of the acquisition unit according to the embodiment. As shown in Fig. 4, the acquisition unit 111 according to the present embodiment includes, for example, an encoded sound information input unit 112, a decoding processing unit 113, a sensing information input unit 114, and a correlation calculation unit 115.

[0092] The encoded sound information input unit 112 is a processing unit to which the coded (in other words, encoded) sound information acquired by the acquisition unit 111 is input. The encoded sound information input unit 112 outputs the input sound information to the decoding processing unit 113. The decoding processing unit 113 is a processing unit that decodes (in other words, decodes) the sound information output from the encoded sound information input unit 112 to generate a playback sound and the position of a sound source object included in the sound information in a format that is used for subsequent processing. The sensing information input unit 114 will be described below together with the function of the detector 103.

[0093] The detector 103 is a device for detecting the speed of movement of the head of the user 99. The detector 103 is configured by combining various sensors used for detecting movement, such as a gyro sensor and an acceleration sensor. In this embodiment, the detector 103 is built into the sound reproduction system 100. However, the detector 103 may be built into an external device, such as a three-dimensional image reproduction device 300 that operates in response to the movement of the head of the user 99 in the same way as the sound reproduction system 100. In this case, the detector 103 does not need to be included in the sound reproduction system 100. Alternatively, the detector 103 may be an external imaging device or the like that captures the movement of the head of the user 99 and detects the movement of the user 99 by processing the captured image.

[0094] The detector 103 is, for example, fixed integrally to the housing of the sound reproduction system 100 and detects the speed of movement of the housing. After the sound reproduction system 100 including the housing is worn by the user 99, it moves integrally with the head of the user 99, and as a result, the detector 103 can detect the speed of movement of the head of the user 99.

[0095] The detector 103 may detect, for example, the amount of head movement of the user 99 as the amount of rotation about at least one of three axes that are orthogonal to each other in three-dimensional space as the rotation axis, or may detect the amount of displacement about at least one of the three axes as the displacement direction. Furthermore, the detector 103 may detect both the amount of rotation and the amount of displacement as the amount of head movement of the user 99.

[0096] The sensing information input unit 114 acquires the movement speed of the user 99's head from the detector 103. More specifically, the sensing information input unit 114 acquires the amount of head movement of the user 99 detected by the detector 103 per unit time as the movement speed. In this way, the sensing information input unit 114 acquires at least one of the rotation speed and the displacement speed from the detector 103. The amount of head movement of the user 99 acquired here is used to determine the position and posture (in other words, coordinates and orientation) of the user 99 in the three-dimensional sound field. Therefore, the acquisition unit 111 also functions as a position acquisition unit via the sensing information input unit 114. In the sound reproduction system 100, the relative position of the sound image object with respect to the user 99 is determined based on the determined coordinates and orientation of the user 99, and sound is reproduced. Specifically, the above functions are realized by the path calculation unit 121 and the output sound generation unit 131.

[0097] The path calculation unit 121 includes an arrival direction calculation function that calculates the relative arrival direction of the reproduced sound from the position of the sound source object to the position of the user 99 based on the determined coordinates and orientation of the user 99, and the conversion process that calculates the secondary sound described above. Therefore, the path calculation unit 121 includes a function that calculates a propagation path from the sound source object and calculates the secondary sound and the arrival direction of the secondary sound that arrives at the position of the user 99 through indirect propagation of the reproduced sound according to the calculated propagation path of the reproduced sound. The arrival direction of the secondary sound includes additional information, such as what object the sound is reflected from in the case of a reflected sound and the attenuation rate at the time of reflection. The additional information is included in the arrival direction of the secondary sound calculated from the input sound information. In other words, the additional information is computationally generated and acquired from the sound information.

[0098] To summarize the spatial information, the spatial information includes further information such as the spatial position of a sound source object in a space (three-dimensional sound field) (information on the position of the sound source object), sound reflection and diffraction characteristics at the sound source object (together with information on the conditions of the spatial environment), and the size of the three-dimensional sound field. Based on the spatial information, the path calculation unit 121 generates secondary sounds depending on which sound source object the reproduced sound is reflected or diffracted by, and calculates, as additional information, the direction from which the secondary sounds arrive and the volume of the secondary sounds after attenuation by reflection or diffraction. The sound information (input data) includes spatial information in the form of audio signals and accompanying metadata. As described above, the spatial information includes, as information other than the audio signal, information necessary to turn the sound into stereophonic sound and position the sound source object in a three-dimensional sound field, and / or information used to calculate information necessary to turn the sound into stereophonic sound and position the sound source object in a three-dimensional sound field.

[0099] The path calculation unit 121 may be realized by any processing as long as it can calculate the arrival direction of the reproduced sound when the reproduced sound reaches the user as a direct sound and can calculate the arrival direction of a secondary sound that arrives at the position of the user 99 due to secondary propagation of the reproduced sound. The path calculation unit 121 determines from which direction in the three-dimensional sound field the reproduced sound and the secondary sound are to be perceived by the user 99 as coming from, based on the coordinates and orientation of the user 99, and processes the sound information so that the output sound signal is perceived as such a sound when it is reproduced.

[0100] The output sound generating unit 131 is a processing unit that processes information about the reproduced sound included in the sound information to generate an output sound signal.

[0101] An example of the output sound generation unit 131 will now be described with reference to Fig. 5. Fig. 5 is a block diagram showing the functional configuration of the output sound generation unit according to an embodiment. As shown in Fig. 5, the output sound generation unit 131 in this embodiment includes, for example, a reduction processing unit 132, which includes a culling unit 133 and an integration unit 134.

[0102] The reduction processing unit 132 is a processing unit that processes sound information by the path calculation unit 121, etc., and determines from multiple sound signals, such as direct sound, reverberant sound, (primary, secondary and higher-order) reflected sound, and indirect sound such as diffracted sound, sound signals that are unlikely to cause an auditory difference even if reduced, i.e., sound signals that are unlikely to cause the user 99 to perceive a deterioration in sound, and reduces those signals.

[0103] The reduction processing unit 132 uses the culling unit 133 to stop the generation of a sound signal, or to perform a culling process that discards the generated sound signal, so that the sound signal is not included in a subsequent output sound signal. The culling unit 133 is a processing unit that discards the specific sound signal determined as the sound to be reduced in this way. Note that discarding here is in a broad sense, including discarding a signal by stopping the generation itself.

[0104] Furthermore, the reduction processing unit 132 discards two or more sound signals using the integrating unit 134, and instead integrates the discarded sound signals into a smaller number of virtual sounds to generate one or more virtual sounds that virtually replace the two or more sounds, so that the two or more sound signals are not included in the subsequent output sound signal, and fewer virtual sound signals are included in the subsequent output sound signal. The integrating unit 134 is a processing unit that discards two or more specific sound signals determined as sounds to be reduced in this way, and generates a smaller number of virtual sound signals to replace them.

[0105] The reduction processing unit 132 determines specific sound signals to be reduced from multiple sound signals based on a preset importance (also expressed as priority) based on the sound source object and the user. For example, the reduction processing unit 132 performs processing such as reducing sound signals whose priority is less than a threshold by culling processing, or reducing two or more sound signals whose priority is less than a threshold by integration processing, and generating and synthesizing virtual sounds that replace the two or more sounds.

[0106] Here, the reduction processing unit 132 performs sound reduction processing according to the relative relationship between the respective states of the sound source object and the user calculated by the relative relationship calculation unit 115 shown in Fig. 4. Specifically, the relative relationship calculation unit 115 calculates a relative relationship at a certain time point (time t) calculated based on at least one of the states of position and orientation between the sound source object at that time point and the user at the same time point (first relative relationship). In addition, the relative relationship calculation unit 115 calculates a relative relationship at that time point (time t + Δt) calculated based on at least one of the states of position and orientation between the sound source object at a subsequent time point that is Δt after the certain time point and the user at the same time point (second relative relationship).

[0107] If the difference in the relative relationship (e.g., difference in position or difference in orientation) between the first relative relationship and the second relative relationship is relatively small, the user 99 is unlikely to notice that the same sound is reduced at these two points in time. In other words, the sound signal can be reduced without causing a sense of incongruity. For example, even if a sound that should be reduced at a subsequent point in time is reduced in advance at a certain point in time, the user 99 is unlikely to feel this sense of incongruity. Therefore, it is easy to reduce more sound signals at a certain point in time, and a significant reduction in the amount of processing can be expected. However, there are cases where it is not appropriate to apply the reduction process described above, such as when the sounds that should be reduced at the two points in time are significantly different. Details of such cases will be explained in more detail in the examples described below.

[0108] Referring again to FIG. 2 , the output sound generation unit 131 acquires a head-related transfer function used to generate an output sound signal from the database 105. The database 105 is an information storage device that functions both as a storage device for storing information and as a storage controller that reads out the stored information and outputs it to an external component. The database 105 stores a head-related transfer function for each direction of arrival of the sound from the user 99. The head-related transfer functions included in the database 105 are a set of general-purpose head-related transfer functions that can be used by everyone, a set of head-related transfer functions optimized for each individual user 99, or a set of head-related transfer functions that are publicly available. The database 105 receives an inquiry from the output sound generation unit 131 using the direction of arrival as a query, and outputs a head-related transfer function corresponding to the direction of arrival to the output sound generation unit 131. In addition, the output sound generation unit 131 may output the entire set of head-related transfer functions, or may output the characteristics of the set of head-related transfer functions itself.

[0109] The signal output unit 141 is a functional unit that outputs the generated output sound signal to the driver 104. The signal output unit 141 generates a waveform signal by performing signal conversion from a digital signal to an analog signal based on the output sound signal, and then causes the driver 104 to generate sound waves based on the waveform signal, thereby presenting sound to the user 99. The driver 104 includes, for example, a diaphragm and a drive mechanism such as a magnet and a voice coil. The driver 104 operates the drive mechanism in response to the waveform signal, causing the drive mechanism to vibrate the diaphragm. In this way, the driver 104 generates sound waves by vibrating the diaphragm in response to the output sound signal (this means "reproducing" the output sound signal; in other words, "reproducing" does not include perception by the user 99). The sound waves propagate through the air and reach the ears of the user 99, and the user 99 perceives the sound.

[0110] [Another Configuration Example] In the above example, the sound reproduction system 100 according to the present embodiment is an audio presentation device, and has been described as including an information processing device 101, a communication module 102, a detector 103, a database 105, and a driver 104. However, the functions of the sound reproduction system 100 may be realized by a plurality of devices or by a single device. Specific examples will be described using Figures 6 to 14. Figures 6 to 14 are diagrams for explaining other examples of the sound reproduction system according to the embodiment.

[0111] For example, the information processing device 601 may be included in the audio presentation device 602, and the audio presentation device 602 may perform both acoustic processing and sound presentation. Alternatively, the information processing device 601 and the audio presentation device 602 may share the acoustic processing described in the present disclosure, or a server connected to the information processing device 601 or the audio presentation device 602 via a network may perform part or all of the acoustic processing described in the present disclosure.

[0112] In the above explanation, the information processing device 601 is referred to as such, but when the information processing device 601 performs acoustic processing by decoding a bitstream generated by encoding at least a portion of the data of an audio signal or spatial information used for acoustic processing, the information processing device 601 may be referred to as a decoding device, and the acoustic reproduction system 100 (i.e., the stereophonic sound reproduction system 600 in the figure) may be referred to as a decoding processing system.

[0113] Here, an example will be described in which the sound reproduction system 100 functions as a decoding processing system.

[0114] <Example of Encoding Device> FIG. 7 is a functional block diagram showing the configuration of an encoding device 700 that is an example of an encoding device according to the present disclosure.

[0115] Input data 701 is data to be coded, including spatial information and / or an audio signal, that is input to an encoder 702. Details of the spatial information will be described later.

[0116] The encoder 702 encodes the input data 701 to generate encoded data 703. The encoded data 703 is, for example, a bit stream generated by the encoding process.

[0117] The memory 704 stores the encoded data 703. The memory 704 may be, for example, a hard disk or a solid-state drive (SSD), or may be any other storage device.

[0118] In the above description, a bitstream generated by an encoding process is given as an example of the encoded data 703 stored in memory 704. However, data other than a bitstream may also be used. For example, the encoding device 700 may convert a bitstream into a predetermined data format and store the converted data in memory 704. The converted data may be, for example, a file or multiplexed stream storing one or more bitstreams. Here, the file may have a file format such as ISOBMFF (ISO Base Media File Format). The encoded data 703 may also be in the form of multiple packets generated by dividing the bitstream or file. When converting the bitstream generated by the encoder 702 into data other than the bitstream, the encoding device 700 may include a conversion unit (not shown), or the conversion process may be performed by a CPU (Central Processing Unit).

[0119] <Example of Decoding Device> FIG. 8 is a functional block diagram showing the configuration of a decoding device 800 that is an example of a decoding device according to the present disclosure.

[0120] The memory 804 stores, for example, the same data as the coded data 703 generated by the coding device 700. The memory 804 reads out the stored data and inputs it as input data 803 to the decoder 802. The input data 803 is, for example, a bitstream to be decoded. The memory 804 may be, for example, a hard disk or an SSD, or may be another storage device.

[0121] Note that the decoding device 800 may not use the data stored in the memory 804 as input data 803 as is, but may convert the read data and generate converted data as input data 803. The data before conversion may be, for example, multiplexed data storing one or more bitstreams. Here, the multiplexed data may be, for example, a file having a file format such as ISOBMFF. The data before conversion may also be in the form of multiple packets generated by dividing the bitstream or file. When converting data different from the bitstream read from the memory 804 into a bitstream, the decoding device 800 may be provided with a conversion unit (not shown), or the conversion process may be performed by a CPU.

[0122] Decoder 802 decodes input data 803 to generate audio signal 801 that is presented to a listener.

[0123] <Another Example of Encoding Device> Fig. 9 is a functional block diagram showing the configuration of an encoding device 900, which is another example of an encoding device of the present disclosure. In Fig. 9, components having the same functions as those in Fig. 7 are assigned the same reference numerals as those in Fig. 7, and descriptions of these components will be omitted.

[0124] The coding device 900 differs from the coding device 700 in that the coding device 700 includes a memory 704 for storing coded data 703, whereas the coding device 900 includes a transmitting unit 901 for transmitting coded data 703 to the outside.

[0125] The transmitter 901 transmits a transmission signal 902 to another device or a server based on the encoded data 703 or data in another data format generated by converting the encoded data 703. The data used to generate the transmission signal 902 is, for example, the bit stream, multiplexed data, file, or packet described in the encoding device 700.

[0126] <Another Example of Decoding Device> Fig. 10 is a functional block diagram showing the configuration of a decoding device 1000, which is another example of a decoding device of the present disclosure. In Fig. 10, components having the same functions as those in Fig. 8 are assigned the same reference numerals, and descriptions of these components will be omitted.

[0127] The decoding device 800 differs from the decoding device 800 in that the decoding device 800 includes a memory 804 for reading out input data 803, whereas the decoding device 1000 includes a receiving unit 1001 for receiving input data 803 from the outside.

[0128] The receiving unit 1001 receives a received signal 1002, acquires received data, and outputs input data 803 to be input to the decoder 802. The received data may be the same as the input data 803 to be input to the decoder 802, or may be data in a data format different from that of the input data 803. If the received data is data in a data format different from that of the input data 803, the receiving unit 1001 may convert the received data into the input data 803, or a conversion unit or CPU (not shown) included in the decoding device 1000 may convert the received data into the input data 803. The received data is, for example, a bit stream, multiplexed data, a file, or a packet, as described in the encoding device 900.

[0129] <Functional Description of Decoder> FIG. 11 is a functional block diagram showing the configuration of a decoder 1100, which is an example of the decoder 802 in FIG. 8 or FIG.

[0130] The input data 803 is an encoded bitstream, and includes encoded audio data, which is an encoded audio signal, and metadata used for acoustic processing.

[0131] The spatial information management unit 1101 acquires metadata included in the input data 803 and analyzes the metadata. The metadata includes information describing elements that act on sounds arranged in a sound space. The spatial information management unit 1101 manages spatial information necessary for acoustic processing obtained by analyzing the metadata and provides the spatial information to the rendering unit 1103. Note that, although the information used for acoustic processing is referred to as spatial information in this disclosure, it may be called by other names. The information used for acoustic processing may be called, for example, sound space information or scene information. Furthermore, when the information used for acoustic processing changes over time, the spatial information input to the rendering unit 1103 may be called a space state, a sound space state, a scene state, or the like.

[0132] Furthermore, spatial information may be managed for each sound space or for each scene. For example, when different rooms are represented as virtual spaces, each room may be managed as a scene of a different sound space, or even if the same space is represented, spatial information may be managed as different scenes depending on the situation being represented. In managing spatial information, an identifier for identifying each piece of spatial information may be assigned. The spatial information data may be included in a bitstream, which is one form of input data 803, or the bitstream may include an identifier for the spatial information, and the spatial information data may be acquired from a source other than the bitstream. When the bitstream includes only the identifier for the spatial information, the identifier for the spatial information may be used during rendering to acquire the spatial information data stored in the memory of the acoustic signal processing device or an external server as input data.

[0133] Note that the information managed by the spatial information management unit 1101 is not limited to information included in the bitstream. For example, the input data 803 may include data indicating the characteristics or structure of a space acquired from a software application or server providing VR or AR, as data not included in the bitstream. Furthermore, for example, the input data 803 may include data indicating the characteristics or position of a listener or object, as data not included in the bitstream. Furthermore, the input data 803 may include, as information indicating the position of the listener, information acquired by a sensor provided in a terminal including a decoding device, or information indicating the position of the terminal estimated based on information acquired by the sensor. In other words, the spatial information management unit 1101 may communicate with an external system or server to acquire spatial information and the position of the listener. Furthermore, the spatial information management unit 1101 may acquire clock synchronization information from an external system and execute a process of synchronizing with the clock of the rendering unit 1103. Note that the space in the above description may be a virtually formed space, i.e., a VR space, or may be a real space or a virtual space corresponding to a real space, i.e., an AR space or an MR (Mixed Reality) space. The virtual space may also be called a sound field or a sound space. Furthermore, the information indicating a position in the above description may be information such as coordinate values ​​indicating a position within a space, information indicating a relative position with respect to a predetermined reference position, or information indicating the movement or acceleration of a position within a space.

[0134] The audio data decoder 1102 decodes the encoded audio data included in the input data 803 to obtain an audio signal.

[0135] The encoded audio data acquired by the stereophonic sound reproduction system 600 is a bitstream encoded in a predetermined format such as MPEG-H 3D Audio (ISO / IEC 23008-3). Note that MPEG-H 3D Audio is merely one example of an encoding method that can be used to generate the encoded audio data included in the bitstream, and the encoded audio data may be included in a bitstream encoded in another encoding method. For example, the encoding method used may be a lossy codec such as MP3 (MPEG-1 Audio Layer-3), AAC (Advanced Audio Coding), WMA (Windows Media Audio), AC3 (Audio Codec-3), or Vorbis, or a lossless codec such as ALAC (Apple Lossless Audio Codec) or FLAC (Free Lossless Audio Codec), or any other encoding method may be used. For example, PCM (Pulse Code Modulation) data may be a type of encoded audio data. In this case, the decoding process may be, for example, a process of converting an N-bit binary number into a number format (e.g., floating-point format) that can be processed by the rendering unit 1103, where the number of quantization bits of the PCM data is N.

[0136] The rendering unit 1103 receives an audio signal and spatial information as input, performs acoustic processing on the audio signal using the spatial information, and outputs an audio signal 801 after acoustic processing.

[0137] Before starting rendering, the spatial information management unit 1101 reads metadata of the input signal, detects rendering items such as objects or sounds defined in the spatial information, and transmits the detected items to the rendering unit 1103. After starting rendering, the spatial information management unit 1101 grasps changes over time in the spatial information and the position of the listener, and updates and manages the spatial information. The spatial information management unit 1101 then transmits the updated spatial information to the rendering unit 1103. The rendering unit 1103 generates and outputs an audio signal to which acoustic processing has been applied based on the audio signal included in the input data and the spatial information received from the spatial information management unit 1101.

[0138] The spatial information update process and the audio signal output process with added acoustic processing may be executed in the same thread, or the spatial information management unit 1101 and the rendering unit 1103 may be allocated to independent threads. When the spatial information update process and the audio signal output process with added acoustic processing are executed in different threads, the thread startup frequency may be set individually, or the processes may be executed in parallel.

[0139] By having the spatial information management unit 1101 and the rendering unit 1103 execute their processes in different, independent threads, computational resources can be preferentially allocated to the rendering unit 1103. Therefore, in the case of sound output processing in which even a slight delay cannot be tolerated, for example, sound output processing in which a delay of even one sample (0.02 msec) would cause a popping noise, can be safely performed. In this case, the allocation of computational resources to the spatial information management unit 1101 is limited. However, compared to audio signal output processing, updating spatial information is a less frequent process (e.g., a process such as updating the listener's facial orientation). Therefore, unlike audio signal output processing, updating spatial information does not necessarily require an instantaneous response, and therefore limiting the allocation of computational resources does not significantly affect the acoustic quality provided to the listener.

[0140] The space information may be updated periodically at preset times or intervals, or when preset conditions are met. The space information may be updated manually by a listener or a sound space manager, or may be triggered by a change in an external system. For example, when a listener operates a controller to instantly warp the position of their avatar, instantly advance or rewind the time, or when a virtual space manager suddenly changes the environment of the space, the thread in which the space information management unit 1101 is located may be started as a one-off interrupt process in addition to being started periodically.

[0141] The role of the information update thread that executes the spatial information update process is, for example, to update the position or orientation of the listener's avatar placed in the virtual space based on the position or orientation of the VR goggles worn by the listener, and to update the position of objects moving in the virtual space. These tasks are handled within a processing thread that runs relatively infrequently, on the order of several tens of Hz. Processing to reflect the properties of direct sound may be performed in such an infrequently occurring processing thread. This is because the properties of direct sound change less frequently than the frequency of audio processing frames for audio output. Doing so can relatively reduce the computational load of the process and also avoid the risk of pulsive noise occurring when information is updated at an unnecessarily fast frequency.

[0142] FIG. 12 is a functional block diagram showing the configuration of a decoder 1200, which is another example of the decoder 802 in FIG. 8 or 10.

[0143] Figure 12 differs from Figure 11 in that the input data 803 includes an unencoded audio signal rather than encoded audio data. The input data 803 includes a bitstream including metadata and an audio signal.

[0144] The spatial information management unit 1201 is the same as the spatial information management unit 1101 in FIG. 11, and therefore a description thereof will be omitted.

[0145] The rendering unit 1202 is the same as the rendering unit 1103 in FIG. 11, and therefore a description thereof will be omitted.

[0146] In the above description, the configuration in Fig. 12 is called a decoder, but it may also be called an audio processing unit that performs audio processing. Furthermore, a device including an audio processing unit may also be called an audio processing device rather than a decoding device. Furthermore, the audio signal processing device (information processing device 601) may also be called an audio processing device.

[0147] <Physical Configuration of Encoding Apparatus> Fig. 13 is a diagram showing an example of the physical configuration of an encoding apparatus. The encoding apparatus shown in Fig. 13 is an example of the encoding apparatuses 700 and 900 described above.

[0148] The encoding device of FIG. 13 includes a processor, a memory, and a communication IF.

[0149] The processor may be, for example, a CPU (Central Processing Unit), a DSP (Digital Signal Processor), or a GPU (Graphics Processing Unit), and the CPU, DSP, or GPU may execute a program stored in a memory to perform the encoding process of the present disclosure. Alternatively, the processor may be a dedicated circuit that performs signal processing on an audio signal, including the encoding process of the present disclosure.

[0150] The memory may be configured, for example, by RAM (Random Access Memory) or ROM (Read Only Memory). The memory may also include a magnetic storage medium such as a hard disk or a semiconductor memory such as an SSD (Solid State Drive). The term "memory" may also refer to an internal memory built into a CPU or GPU.

[0151] The communication IF (Interface) is a communication module compatible with a communication method such as Bluetooth (registered trademark) or WIGIG (registered trademark). The encoding device has a function of communicating with other communication devices via the communication IF and transmits an encoded bitstream.

[0152] The communication module is composed of, for example, a signal processing circuit and an antenna corresponding to the communication method. In the above example, Bluetooth (registered trademark) or WIGIG (registered trademark) is used as the communication method, but other communication methods such as LTE (Long Term Evolution), NR (New Radio), or Wi-Fi (registered trademark) may also be supported. Furthermore, the communication IF may be a wired communication method such as Ethernet (registered trademark), USB (Universal Serial Bus), or HDMI (registered trademark) (High-Definition Multimedia Interface) instead of the wireless communication method described above.

[0153] <Physical configuration of audio signal processing device> Fig. 14 is a diagram showing an example of the physical configuration of an audio signal processing device. Note that the audio signal processing device in Fig. 14 may be a decoding device. Furthermore, part of the configuration described here may be provided in the audio presentation device 602. Furthermore, the audio signal processing device shown in Fig. 14 is an example of the audio signal processing device 601 described above.

[0154] The acoustic signal processing device of FIG. 14 includes a processor, a memory, a communication IF, a sensor, and a speaker.

[0155] The processor may be, for example, a CPU (Central Processing Unit), a DSP (Digital Signal Processor), or a GPU (Graphics Processing Unit), and the CPU, DSP, or GPU may execute a program stored in a memory to perform the audio processing or decoding processing of the present disclosure. Alternatively, the processor may be a dedicated circuit that performs signal processing on audio signals, including the audio processing of the present disclosure.

[0156] The memory may be configured, for example, by RAM (Random Access Memory) or ROM (Read Only Memory). The memory may also include a magnetic storage medium such as a hard disk or a semiconductor memory such as an SSD (Solid State Drive). The term "memory" may also refer to an internal memory built into a CPU or GPU.

[0157] The communication IF (Interface) is a communication module compatible with a communication method such as Bluetooth (registered trademark) or WIGIG (registered trademark). The acoustic signal processing device shown in FIG. 2I has a function of communicating with other communication devices via the communication IF and acquires a bitstream to be decoded. The acquired bitstream is stored in a memory, for example.

[0158] The communication module is composed of, for example, a signal processing circuit and an antenna corresponding to the communication method. In the above example, Bluetooth (registered trademark) or WIGIG (registered trademark) is used as the communication method, but other communication methods such as LTE (Long Term Evolution), NR (New Radio), or Wi-Fi (registered trademark) may also be supported. Furthermore, the communication IF may be a wired communication method such as Ethernet (registered trademark), USB (Universal Serial Bus), or HDMI (registered trademark) (High-Definition Multimedia Interface) instead of the wireless communication method described above.

[0159] The sensor performs sensing to estimate the position or orientation of the listener. Specifically, the sensor estimates the position and / or orientation of the listener based on one or more detection results of the position, orientation, movement, velocity, angular velocity, or acceleration of a part or the entire body of the listener, such as the head, and generates position information indicating the position and / or orientation of the listener. Note that the position information may be information indicating the position and / or orientation of the listener in real space, or information indicating a displacement of the position and / or orientation of the listener based on the position and / or orientation of the listener at a predetermined time. Furthermore, the position information may be information indicating the position and / or orientation relative to the stereophonic sound reproduction system or an external device equipped with the sensor.

[0160] The sensor may be, for example, an imaging device such as a camera or a ranging device such as LiDAR (Light Detection and Ranging), and may capture an image of the listener's head movement and detect the movement of the listener's head by processing the captured image. Alternatively, the sensor may be a device that performs position estimation using wireless signals of any frequency band, such as millimeter waves.

[0161] The audio signal processing device shown in Fig. 14 may acquire position information from an external device equipped with a sensor via a communication IF. In this case, the audio signal processing device does not need to include a sensor. Here, the external device is, for example, the audio presentation device 602 described in Fig. 6 or a 3D video playback device worn on the listener's head. In this case, the sensor is configured by combining various sensors such as a gyro sensor and an acceleration sensor.

[0162] The sensor may, for example, detect the angular velocity of rotation around at least one of three mutually perpendicular axes in the sound space as the axis of rotation as the speed of movement of the listener's head, or may detect the acceleration of displacement with at least one of the three axes as the direction of displacement.

[0163] For example, the sensor may detect the amount of rotation about at least one of three mutually orthogonal axes in the sound space as the rotation axis, or the amount of displacement about at least one of the three axes as the displacement direction, as the amount of movement of the listener's head. Specifically, the sensor detects 6 DoF (position (x, y, z) and angle (yaw, pitch, roll)) as the position of the listener. The sensor is configured by combining various sensors used for detecting movement, such as a gyro sensor and an acceleration sensor.

[0164] The sensor may be any device capable of detecting the position of the listener, such as a camera or a GPS (Global Positioning System) receiver. Alternatively, the sensor may use location information obtained by performing self-position estimation using LiDAR (Laser Imaging Detection and Ranging). For example, when the audio signal reproduction system is implemented by a smartphone, the sensor is built into the smartphone.

[0165] The sensor may also include a temperature sensor such as a thermocouple that detects the temperature of the acoustic signal processing device shown in Figure 14, and a sensor that detects the remaining charge of a battery provided in or connected to the acoustic signal processing device.

[0166] A speaker has, for example, a diaphragm, a drive mechanism such as a magnet or a voice coil, and an amplifier, and presents an audio signal after acoustic processing to a listener as sound. The speaker operates the drive mechanism in response to the audio signal (more specifically, a waveform signal indicating the waveform of the sound) amplified by the amplifier, and the drive mechanism vibrates the diaphragm. In this way, the diaphragm vibrating in response to the audio signal generates sound waves, which propagate through the air and reach the listener's ears, causing the listener to perceive the sound.

[0167] Note that, although the description has been given here of an example in which the acoustic signal processing device shown in FIG. 14 includes a speaker and presents an audio signal after acoustic processing via the speaker, the means for presenting the audio signal is not limited to the above configuration. For example, the audio signal after acoustic processing may be output to an external audio presentation device 602 connected via a communication module. Communication via the communication module may be wired or wireless. As another example, the acoustic signal processing device shown in FIG. 14 may include a terminal for outputting an analog audio signal, and a cable such as earphones may be connected to the terminal to present the audio signal from the earphones. In the above case, the audio signal is reproduced by headphones, earphones, a head-mounted display, a neck speaker, a wearable speaker, a surround speaker composed of multiple fixed speakers, or the like, which is worn on the head or part of the body of the listener, which is the audio presentation device 602.

[0168] <Functional Description and Examples of Rendering Unit> Hereinafter, detailed configurations of the rendering units 1103 and 1202 in Fig. 11 and Fig. 12 will be described using examples. Figs. 15 to 33 are diagrams for explaining specific examples of sound reproduction systems according to examples of the embodiment.

[0169] In this embodiment, when reducing sound for a listener (user 99) whose position and orientation are at time t, sound is reduced according to the relative relationship of the listener's position and orientation to the position and orientation of the sound source object at the same time (hereinafter also referred to as the relative relationship of the state), and sound is also reduced according to the relative relationship of the listener's position and orientation to the position and orientation of the sound source object at time t + Δt by calculating, for example, predicting the listener's movement a time Δt later.

[0170] For example, calculations are made by predicting the listener's position a time Δt from the movement history up to now, and sounds that will be below the loudness threshold at a position a time Δt from now are removed from candidates for sounds that will not be reduced at time t, i.e., reduced. This is based on the idea that sounds that will become inaudible after a time Δt are considered to have little impact even at time t, and sounds that will become inaudible at the estimated position at time t + Δt are also reduced at time t, thereby suppressing deterioration in sound quality while achieving the effect of reducing the amount of processing.

[0171] FIG. 15 will be described. FIG. 15 is a block diagram of a decoder (i.e., a rendering unit 1500 and a spatial information management unit 1501) according to this example. The basic idea of ​​this example is that sounds that become inaudible after a time Δt (hereinafter, Δt is a time, such as milliseconds to several seconds, in which a listener will not easily perceive a difference, and is also expressed as after Δt seconds) are considered to have little impact, and sounds that become inaudible at the estimated position at time t + Δt are also reduced at time t, thereby reducing the amount of calculation. In the following example, culling processing will be mainly described as an example of reduction processing, but the reduction processing in this example can be at least either culling processing or integration processing, and is not limited to culling processing.

[0172] First, input data (such as a bit stream) is provided to the spatial information management unit 1501. The input data includes an audio signal or encoded audio data representing an audio signal, and metadata used in acoustic processing. If encoded audio data is included, the encoded audio data is provided to an audio data decoder (not shown), which performs decoding processing and generates an audio signal. This audio signal is provided to the direct sound generation unit 1502, the reverberation sound generation unit 1503, the reflected sound generation unit 1504, and the diffracted sound generation unit 1505. If an audio signal is included instead of encoded audio data, the audio signal is provided directly to these sound generation units (i.e., the direct sound generation unit 1502, the reverberation sound generation unit 1503, the reflected sound generation unit 1504, and the diffracted sound generation unit 1505; the same applies hereinafter when simply referred to as the "sound generation unit" or the "generation unit").

[0173] The spatial information management unit 1501 extracts metadata from the input data, and provides the metadata to the direct sound generation unit 1502, the reverberation sound generation unit 1503, the reflected sound generation unit 1504, and the diffracted sound generation unit 1505. The metadata configuration is shown in Fig. 16. The spatial information 1601 mainly represents information about the space in which immersive audio is provided to the listener, such as the shape of the room, the characteristics of the wall materials (such as sound reflectivity and absorption rate), the characteristics of the materials of obstacles (such as sound reflectivity and absorption rate), and information about their placement.

[0174] The object information 1602 mainly represents information relating to the position and orientation of the sound source object and the sound emitted from the sound source object.

[0175] The listener information 1603 mainly represents information relating to the position and orientation of the listener.

[0176] The direct sound generation unit 1502, reverberation sound generation unit 1503, reflected sound generation unit 1504, and diffracted sound generation unit 1505 each receive an audio signal and metadata, generate direct sound, reverberation sound, reflected sound, and diffracted sound, and output them to culling units 1506a to 1506d. The decoder in Figure 15 has each generation unit configured in parallel, and is configured to cull the sounds generated by each generation unit. Here, culling units 1506a to 1506d are arranged downstream of all of the generation units, but this is not limiting, and a configuration in which a culling unit is not arranged downstream of some of the generation units may also be used.

[0177] Each of the culling units 1506a to 1506d identifies unimportant sounds from the input signal, discards the identified sounds, and outputs the remaining sounds to the sound generation unit 1507. Discarding sounds can also be expressed as bypassing or ignoring them.

[0178] The sound generation unit 1507 performs sound processing such as HRTF (Head related transfer function) convolution processing on the signals input from each of the culling units 1506a to 1506d, and outputs the signals as output sound signals (also referred to as output signals). This sound processing is performed to suit the output format to the listener, such as headphones or multi-channel speakers, and the output signals are provided to the listener.

[0179] 17 will be described. Fig. 17 is a diagram showing the concept of the culling process in this example. This diagram shows an example in which the position of the sound source object 98 does not move, but the position of the listener (user 99) moves.

[0180] Assume that the following sounds reach the listener's position at time t: (a) direct sound, (b) reflected sound, (c) to (g) reverberation sounds, and (h) diffracted sound, as shown in the figure. Of the sounds that reach the listener, sounds with low importance according to a certain reference value are culled. In the figure, the cross arrows represent sounds that are culled at time t. Note that the reference value here may be the energy of the sound that reaches the listener or the amount of change in energy over time, or it may be one that utilizes the listener's auditory characteristics. In the following, as an example, the magnitude of sound energy corresponds to importance.

[0181] Next, the position of the listener Δt seconds later (i.e., time t + Δt) is predicted. To predict the listener's position, past listener positions are stored in a queue (first-in-first-out memory, FIFO), and a prediction is made based on the listener position information stored in the queue, thereby calculating the listener's position at time t + Δt. The prediction method may be linear prediction, or it may be a simple extrapolation of the most recent position information.

[0182] In order to obtain the exact listener position Δt seconds later, a slight delay may be tolerated and the next metadata may be input, and the listener information contained in that metadata may be used. Specifically, if the nth metadata is represented as Meta(n), the system waits for Meta(n+1) to be input, obtains the listener position information contained in Meta(n+1), and then processes Meta(n). Therefore, although there is a delay of Δt seconds until the next metadata is input, it is possible to obtain the exact listener position Δt seconds later.

[0183] Next, the energy of each sound at the listener's position at time t + Δt is calculated. Of the sounds that reach the listener, sounds with low importance according to a certain reference value are culled. In the figure, rectangular arrows represent sounds that are culled at time t + Δt.

[0184] To reduce the amount of calculation, the energy of each sound is calculated by predicting it from the energy of each sound calculated at the listener's position at time t. The prediction method will be described later.

[0185] Note that, while it is assumed here that the energy of the sound reaching the listener at time t+Δt is calculated, this is not limiting, and sounds to be culled may be determined based on whether the listener is moving rapidly. In other words, if the listener's position is significantly different between time t and time t+Δt, it is assumed that the listener's sound environment has changed significantly, and sounds with low importance to the listener (sounds with low importance set for the listener, such as reflected sounds, reverberation sounds, and diffracted sounds) are culled. This eliminates the need to calculate the sounds that reach the listener at time t+Δt, thereby reducing the amount of calculation required.

[0186] Next, flowcharts of the operation of the decoder in the above configuration are shown in Figures 18 and 19. Note that Figure 19 shows a detailed flow of step S1803 in Figure 18.

[0187] First, it is determined whether metadata has been input (S1801). If metadata has been input (Yes in S1801), direct sound is generated (S1802). If metadata has not been input (No in S1801), the process ends.

[0188] After generating the direct sound (S1802), it is determined whether culling of the most recently generated sound (i.e., the direct sound) is performed (S1803). For example, it is determined whether the flag is 1 (S1803a). If the flag is 1 (Yes in S1803a), normal culling based only on time t is performed (S1803b), and the process proceeds to generating the next generated sound (i.e., the reverberant sound). If the flag is not 1 (No in S1803a), it is determined whether the flag is 2 (S1803c). If the flag is 2 (Yes in S1803c), culling based on both time t and time t + Δt is performed (S1803d), and the process proceeds to generating the next generated sound (i.e., the reverberant sound). If the flag is not 2 (No in S1803c), the flag is determined to be 0, and the process proceeds to generating the next generated sound (i.e., the reverberant sound) without performing culling.

[0189] Note that this flag may always be included in the metadata, or may be set in advance as a default value. Alternatively, it may be set in advance as a default value, and when an updated value for the flag is included in the metadata, the flag may be updated to that value and continued to be used. Also, a flag for each sound generation unit may be set independently. The same applies to flags that appear in the following flowcharts.

[0190] 18, reverberation is generated (S1804), and it is determined whether or not culling of the most recently generated sound (i.e., reverberation) is to be performed, as described above (S1803). For reverberation, as well, based on the flag value of 0, 1, or 2, either no culling is performed, culling is performed taking into account only time t, or culling is performed taking into account both time t and time t+Δt is performed.

[0191] 18, reflected sounds are generated (S1805), and it is determined whether or not culling of the most recently generated sound (i.e., reflected sounds) is to be performed, as described above (S1803). For reflected sounds, as well, based on the flag value of 0, 1, or 2, either no culling is performed, culling is performed taking into account only time t, or culling is performed taking into account both time t and time t+Δt is performed.

[0192] 18, diffracted sound is generated (S1806), and it is determined whether or not culling of the most recently generated sound (i.e., diffracted sound) is to be performed, as described above (S1803). For diffracted sound as well, based on the flag value of 0, 1, or 2, either no culling is performed, culling is performed taking into account only time t, or culling is performed taking into account both time t and time t+Δt is performed.

[0193] As a result of the above-mentioned generation of generated sounds and culling processing, the remaining direct sound, reverberant sound, reflected sound, and diffracted sound are subjected to stereophonic signal processing such as HRTF convolution to generate a stereophonic signal (S1807), which is then output to a device used by the listener, such as headphones.

[0194] Then, the process returns to step S1801, where it is determined whether new metadata is to be input.

[0195] Note that, here, a method is described in which one of three options is selected: when no culling is performed, when conventional culling based only on time t is performed, and when culling based on both time t and time t+Δt, which is a feature of this example, but the method is not limited to this, and it may also be a method in which one of two options is selected: when no culling is performed or when culling based on both time t and time t+Δt is performed. In this case, the flag indicates 0 or 1.

[0196] Referring now to Figure 20, Figure 20 is a block diagram of another configuration of a decoder according to this example. This configuration is characterized in that a direct sound generation unit 1502, a reverberation sound generation unit 1503, a reflected sound generation unit 1504, and a diffracted sound generation unit 1505 are connected in series, and culling units 1506a to 1506d are arranged between each generation unit. With this configuration, the sound generated by the generation unit in the previous stage can affect the generation unit in the current stage, making it possible to provide accurate immersive audio that is closer to actual spatial audio.

[0197] In this figure, one of the culling units 1506a to 1506d is placed between all of the generation units, but this is not limited to this, and a configuration in which some of the culling units are not placed may also be used.

[0198] Next, we will explain Fig. 21. Fig. 21 shows a diagram similar to Fig. 17. Here, we will explain a method for estimating and determining the sound at time t+Δt from the sound at time t in order to reduce the amount of calculation. Note that for ease of explanation, only the reflected sound (b) is used for the explanation, but the same can be said for the direct sound (a), the reverberant sounds (c) to (g), and the diffracted sound (h).

[0199] First, the position of the reference point is calculated from the position L(t) of the listener at time t and the position of the sound source object 98. From the position of the reference point and the position of the listener, distances d(t) and d(t+Δt) are calculated. Here, d(t) represents the distance between the reference point and the position of the listener at time t. When the energy of the sound heard by the listener at time t is E(t), the energy E(t+Δt) of the sound at time t+Δt is calculated according to the following equation (1).

[0200]

[0201] Note that this formula (1) is derived from the relationship that sound pressure is inversely proportional to the square of the distance, but if a different relationship is assumed, that relationship can be used to derive E(t+Δt).The calculated E(t+Δt) is then compared with the loudness threshold.If E(t+Δt) is below the threshold, the reflected sound to the listener at time t is culled.

[0202] In the case of a direct sound, the position of the reference point is not calculated, and the position of the sound source object 98 is regarded as the position of the reference point, and calculation processing is performed in the same manner as above.

[0203] Now, let us consider Figure 22. Figure 22 shows a diagram similar to Figure 17. The basic idea in this example is the same as above, that sounds that will no longer be heard after Δt seconds are considered to have little impact, and sounds that will no longer be heard at the estimated position at time t + Δt are also culled at time t, thereby reducing the amount of calculation.

[0204] While in the above example, sounds to be culled are determined by predicting the movement of the listener, this example is characterized in that sounds to be culled are determined by predicting the movement of the sound source object 98. Figure 22 shows a conceptual diagram of the culling process in this example.

[0205] Assume that the following sounds reach the listener's position at time t: (a) direct sound, (b) reflected sound, (c)-(g) reverberation sounds, and (h) diffracted sound, as shown in the figure. Of the sounds that reach the listener, sounds with low importance according to a certain reference value are culled. In the figure, the cross arrows represent the sounds that are culled at time t.

[0206] Next, the position of the sound source object 98 after Δt seconds (i.e., time t + Δt) is predicted. To predict the position of the sound source object 98, past positions of the sound source object 98 are stored in a queue, and prediction is performed based on the position information of the sound source object 98 stored in the queue to determine the position of the sound source object 98 at time t + Δt. As a prediction method, linear prediction may be used, or prediction may be made by simply extrapolating the most recent position information.

[0207] In order to obtain the accurate position of the sound source object 98 Δt seconds later, a slight delay may be tolerated and the next metadata may be input, and the information of the sound source object 98 contained in that metadata may be used. Specifically, if the n-th metadata is represented as Meta(n), the system waits for Meta(n+1) to be input, obtains the position information of the sound source object 98 contained in Meta(n+1), and then processes Meta(n). Therefore, although there is a delay until the next metadata is input, it is possible to obtain the accurate position of the sound source object 98 Δt seconds later.

[0208] Next, the energy of each sound at the position of the sound source object 98 at time t+Δt is calculated. Of the sounds that reach the listener, sounds with low importance according to a certain reference value are culled. In the figure, rectangular arrows represent sounds that are culled at time t+Δt.

[0209] To reduce the amount of calculation, the energy of each sound is calculated by predicting it from the energy of each sound calculated at the listener's position at time t. The method of prediction will be described later.

[0210] Note that, while it is assumed here that the energy of the sound that reaches the listener at time t + Δt is calculated, this is not limiting, and sounds to be culled may be determined based on whether the movement of the sound source object 98 is rapid. In other words, if the position of the sound source object 98 at time t is significantly different from that at time t + Δt, it is assumed that the sound environment of the listener has changed significantly, and sounds that are less important to the listener (for example, reflected sounds, reverberation sounds, and diffracted sounds) are culled. This eliminates the need to calculate the sounds that reach the listener at time t + Δt, thereby suppressing an increase in the amount of calculations.

[0211] Next, we will explain Fig. 23. Fig. 23 shows a diagram similar to Fig. 17. Here, we will explain a method for estimating and determining the sound at time t+Δt from the sound at time t in order to reduce the amount of calculation. Note that, for ease of explanation, only the reflected sound (b) is used for the explanation, but the same can be said for the direct sound (a), the reverberant sounds (c) to (g), and the diffracted sound (h).

[0212] First, the position of the reference point is calculated from the position O(t) of the sound source object 98 at time t and the position of the listener. Distances d(t) and d(t+Δt) are calculated from the positions of the reference point and the position of the sound source object 98. Here, d(t) represents the distance between the position of the sound source object 98 at time t and the reference point. From the energy E(t) of the sound heard by the listener at time t, the energy E(t+Δt) of the sound at time t+Δt is calculated according to the above equation (1). Then, the calculated E(t+Δt) is compared with the loudness threshold. If E(t+Δt) is below the threshold, the sound to the listener at time t is culled.

[0213] In the case of direct sound, the position of the reference point is not calculated, and the position of the listener is regarded as the position of the reference point, and calculation processing is performed in the same manner as above.

[0214] FIG. 24 will now be described. FIG. 24 shows a diagram similar to FIG. 17. The following example is characterized in that it determines whether to perform culling of sounds reaching the listener, taking into consideration the change in the direction of the listener's face (azimuth angle: θ, elevation angle: φ) after Δt seconds. The direction of the listener's face is also determined by prediction. Note that, for ease of explanation, only the reflected sound (b) is used for the explanation here, but the same can be said for the direct sound (a), the reverberated sounds (c) to (g), and the diffracted sound (h).

[0215] The method of predicting the listener's face direction is similar to that of predicting the listener's position, in which the listener's past face directions are stored in a queue and linear prediction is used based on the listener's past face directions stored in the queue, or a prediction can be made by extrapolating the most recent face direction.

[0216] In order to obtain the exact facial direction of the listener Δt seconds later, a configuration may be adopted in which a slight delay is tolerated and the next metadata is input, and the listener information included in the metadata is used. Specifically, if the nth metadata is represented as Meta(n), the configuration waits for Meta(n+1) to be input, obtains facial direction information from the listener information included in Meta(n+1), and then processes Meta(n). Therefore, although there is a delay until the next metadata is input, it is possible to obtain the exact facial direction of the listener Δt seconds later. Figure 24 shows a conceptual diagram of the culling process in this example.

[0217] If the sound energy reaching a listener when the listener's face is oriented in a direction (θ(t), φ(t)) at time t is defined as E(t), the sound energy reaching the listener when the listener's face is oriented in a direction (θ(t+Δt), φ(t+Δt)) at time t+Δt, E(t+Δt), is calculated according to the following formula (2).

[0218]

[0219] This energy estimate E(t+Δt) at time t+Δt is compared with the loudness threshold, and if E(t+Δt) is below the threshold, the reflected sound to the listener at time t is culled.

[0220] Note that, while it is assumed here that the energy of the sound reaching the listener at time t + Δt is calculated, this is not limiting, and sounds to be culled may be determined based on whether the direction of the listener's face changes suddenly. In other words, if the direction of the listener's face changes significantly between time t and time t + Δt, it is assumed that the listener's sound environment has changed significantly, and sounds that are less important to the listener (for example, reflected sounds, reverberation sounds, and diffracted sounds) are culled. This eliminates the need to calculate the sounds that reach the listener at time t + Δt, making it possible to suppress an increase in the amount of calculations.

[0221] FIG. 25 will now be described. FIG. 25 shows a diagram similar to FIG. 17. The following example shows another implementation of the example described in FIG. 24. This example is characterized in that the decision as to whether to perform culling of a sound is made taking into consideration the change in the direction (azimuth angle: θ, elevation angle: φ) of the sound source object 98 after Δt seconds. Note that, for ease of explanation, only the reflected sound (b) is used for the explanation, but the same can be said for the direct sound (a), the reverberant sounds (c) to (g), and the diffracted sound (h).

[0222] In addition, the orientation of the sound source object 98 is determined by prediction, and the method of predicting the orientation of the sound source object 98 is similar to the method of predicting the position of the sound source object 98, in that past orientations of the sound source object 98 are stored in a queue, and linear prediction may be used based on the past orientations of the sound source object 98 stored in the queue, or the orientation of the most recent sound source object 98 may be extrapolated to make a prediction.

[0223] In order to obtain the accurate orientation of the sound source object 98 Δt seconds later, a slight delay may be tolerated and the next metadata may be input, and the information of the sound source object 98 included in that metadata may be used. Specifically, if the n-th metadata is represented as Meta(n), the system waits for Meta(n+1) to be input, obtains information on the orientation of the sound source object 98 from the information of the sound source object 98 included in Meta(n+1), and then processes Meta(n). Therefore, although there is a delay until the next metadata is input, it is possible to obtain the accurate orientation of the sound source object 98 Δt seconds later. Figure 25 shows a conceptual diagram of the culling process in this example.

[0224] If the sound energy reaching the listener when the direction of the sound source object 98 at time t is (θ(t), φ(t)) is E(t), the sound energy reaching the listener when the direction of the sound source object 98 at time t + Δt is (θ(t + Δt), φ(t + Δt)) is calculated according to the above formula (2).

[0225] This energy estimate E(t+Δt) at time t+Δt is compared with the loudness threshold, and if E(t+Δt) is below the threshold, the reflected sound to the listener at time t is culled.

[0226] Note that, while it is assumed here that the energy of the sound that reaches the listener at time t + Δt is calculated, this is not limiting, and sounds to be culled may be determined based on whether the direction of the sound source object 98 changes suddenly. In other words, if the direction of the sound source object 98 changes significantly between time t and time t + Δt, it is assumed that the sound environment of the listener has changed significantly, and sounds that are less important to the listener (for example, reflected sounds, reverberation sounds, and diffracted sounds) are culled. This eliminates the need to calculate the sounds that reach the listener at time t + Δt, making it possible to suppress an increase in the amount of calculations.

[0227] 26 will be described. This example is characterized in that the position change and the orientation change of the listener or sound source object 98 are estimated at different time resolutions (note that different time intervals for updating the position and the orientation change are referred to as different time resolutions here). This is based on the idea that, because orientation changes tend to occur more frequently than position changes, more frequent reflection of orientation changes will improve culling performance.

[0228] In some applications, position changes occur more frequently than orientation changes. In such cases, culling performance can be improved by increasing the time resolution of position changes compared to orientation changes.

[0229] 26 is a conceptual diagram showing the timing of metadata acquisition according to this example. The horizontal axis represents time, and the vertical lines represent the timing of metadata reception, with the queue being updated each time metadata is received. The timing of position estimation (culling based on position information) is represented by open triangles, and the timing of orientation estimation (culling based on orientation information) is represented by open circles. This diagram shows that orientation change estimation is performed with a higher time resolution than position change estimation.

[0230] Next, FIG. 27 will be described. FIG. 27 shows a diagram similar to FIG. 17. A feature of this example is that, if the movement of the listener or sound source object 98 is discontinuous, the culling based on motion prediction as described above is stopped. This means that, if the movement is discontinuous, the environment surrounding the listener at time t and time t + Δt is significantly different. Therefore, performing culling at time t using information from time t + Δt may result in the reduction of sounds that are actually necessary due to culling, which may result in the listener receiving unnatural immersive audio. Therefore, by using the method described in this example, such problems can be avoided.

[0231] The sound referred to here may be any of direct sound, reflected sound, reverberation sound, and diffracted sound.

[0232] Depending on the application, the movement of the listener or sound source object 98 may be discontinuous (for example, in a game using immersive audio, the player may warp to another location after completing a certain event). In such cases, the execution of culling based on predicted movement is stopped.

[0233] 27 shows a conceptual diagram of discontinuous movement assumed in this example. If the listener's position L(T) at time t and the listener's position L(t+Δt) Δt seconds later are significantly different, the listener's environment will change significantly. Therefore, if culling at time t taking time t+Δt into consideration is performed, necessary sounds may also be culled, potentially causing problems such as a deterioration in the sound effect. In this way, when the user's movement is discontinuous, culling based on motion prediction is stopped to avoid the above-mentioned problems.

[0234] One method for determining whether the listener's movement is discontinuous is to determine that the movement is discontinuous if the amount of movement of the listener exceeds a predetermined threshold. Also, if the metadata includes a flag indicating that the listener's movement is discontinuous, it is possible to determine whether the listener's movement is discontinuous by referring to the flag.

[0235] Although the movement of the listener has been described as an example here, the present invention is not limited to this, and the same can be said for cases where the movement of the sound source object 98 is discontinuous.

[0236] Next, FIG. 28 will be described. FIG. 28 shows a diagram similar to FIG. 17 . This example is characterized in that, if an obstacle 97 appears on the sound path between the sound source object 98 and the listener due to movement of the listener or sound source object 98, the culling process based on motion prediction is stopped. This means that if an obstacle 97 appears in the sound path at time t + Δt, the environment surrounding the listener at time t and time t + Δt is significantly different. Therefore, performing culling at time t using information from time t + Δt may result in the culling of sounds that are actually necessary, which may result in the listener experiencing unnatural immersive audio. Therefore, by using the present example, such problems can be avoided.

[0237] The sound referred to here may be any of direct sound, reflected sound, reverberation sound, and diffracted sound.

[0238] Fig. 28 shows a conceptual diagram of the occurrence of an obstacle assumed in this example. Fig. 28 illustrates a case where an obstacle 97 occurs in the path of the direct sound (a) due to the movement of the listener. As the environment surrounding the listener changes significantly as the listener moves, the culling process based on motion prediction is stopped. Whether an obstacle 97 will occur in such a sound path can be determined from the information contained in the metadata itself or by analyzing that information.

[0239] Although the movement of the listener has been described as an example here, the present invention is not limited to this, and the same can be said for the case where the sound source object 98 moves.

[0240] Alternatively, it may be possible to determine whether the movement of the listener or sound source object 98 is fast, and similarly stop the culling process based on motion prediction if the movement is fast. If the movement of the listener or sound source object 98 is fast, there is a high possibility that an obstacle 97 will appear between the sound source object 98 and the listener. Therefore, by stopping the culling process based on motion prediction, it is possible to avoid providing the listener with unnatural immersive audio, as described above.

[0241] The above description can be considered a simplified version of the example of detecting the occurrence of an obstacle 97, where instead of determining whether an obstacle 97 appears in the path of the sound, the determination is made based on the speed of movement of the listener or sound source object 98.

[0242] The speed of movement of the listener or sound source object 98 can be calculated by storing the past positions of the listener or sound source object 98 in a queue and using the position information of the listener or sound source object 98 stored in the queue.

[0243] 29 and 30 will be described. FIG. 29 shows a diagram similar to FIG. 17. The above-described embodiment is characterized in that the movement of a listener Δt seconds from now is predicted for a listener located at time t, and culling of sounds that will reach the listener at time t is also performed for sounds that will reach the listener at time t + Δt. In contrast, this example is characterized in that the movement of a listener n×Δt seconds from now (n is an integer of 2 or greater) from a listener located at time t is predicted, and culling of sounds that will reach the listener at time t is also performed for sounds that will reach the listener at positions from time t + Δt to time t + n Δt. This example estimates the position of the listener at a time further in the future than the previously described embodiments, and culls sounds that have a small impact among sounds that will reach the listener up to that future time. This makes it possible to further increase the number of sounds that will be subject to culling, thereby making it possible to more significantly reduce the amount of calculation.

[0244] 29 and 30 show conceptual diagrams of the culling process in this example. These diagrams show an example in which the position of the sound source object 98 does not move, but the position of the listener moves.

[0245] Assume that the following sounds reach the listener's position at time t: (a) direct sound, (b) reflected sound, (c) to (f) reverberation sounds, and (h1) to (h4) diffracted sound, as shown in the figure. Of the sounds reaching the listener, sounds with low importance are culled according to a certain reference value. In the figure, the cross arrow represents the sound to be culled at time t, the rectangular arrow represents the sound to be culled at time t + Δt, the black circle arrow represents the sound to be culled at time t + 2 Δt, and the white circle arrow represents the sound to be culled at time t + 3 Δt. Note that the reference value here may be the energy of the sound reaching the listener or the amount of change in energy over time, or it may be one that utilizes the listener's auditory characteristics. The listener's position at each time is predicted and determined using the method already described.

[0246] The figure shows an example in which the diffracted sound (h4) at time t, the diffracted sound (h3) at time t + Δt, the reverberated sound (e) at time t + 2 Δt, and the reverberated sound (f) and the diffracted sound (h1) at time t + 3 Δt are identified as the sounds to be culled.

[0247] The time up to which these culled sounds have an effect is shown in Figure 30. As shown in Figure 30, sounds identified as being culled at a certain time are culled going back to a predetermined past time, thereby achieving an efficient reduction in the amount of calculations. This is a feature of this example.

[0248] In the above example, the case where the listener moves has been described, but the same can be said for the case where the sound source object 98 moves.

[0249] Next, FIG. 31 will be described. FIG. 31 shows a diagram similar to FIG. 17. In the above-described embodiment, the movement of the listener Δt seconds after the listener is located at time t is predicted, and the position of the listener at time t + Δt is also used for culling at time t. In contrast, this example is characterized in that it determines whether the listener's movement is toward or away from the sound reaching the listener, and culling is performed based on the determination result. If the listener's movement is toward or away from the sound reaching the listener, and the reference value of the sound is compared with a predetermined threshold and is below the threshold, the sound at time t + Δt and thereafter is always culled. This is an effective method for scenes where the listener's movement is facing in a substantially constant direction. Note that the reference value here may be the energy of the sound reaching the listener or the amount of change in energy over time, or it may be one that utilizes the listener's auditory characteristics.

[0250] Figure 31 shows a conceptual diagram of the listener's movement in this example. This diagram depicts a scene in which the listener is moving toward the exit in the upper right corner. Up until time t, the listener is moving toward the reflected sound (b), and from time t + Δt onwards, the listener is moving away from the reflected sound (b). Note that while the explanation here uses the reflected sound (b) as an example, it is not limited to this, and the concept can also be applied to the direct sound (a), the reverberant sounds (c) to (g), and the diffracted sound (h).

[0251] Up to time t, the reflected sound (b) is not culled because the listener is moving in a direction approaching the reflected sound (b). On the other hand, after time t + Δt, the listener is moving in a direction away from the reflected sound (b). Therefore, after time t + Δt, the reflected sound (b) becomes a candidate for culling. If the reflected sound (b) becomes a candidate for culling, a reference value for the reflected sound (b) is compared with a predetermined threshold, and if the reference value is below the predetermined threshold, the reflected sound (b) is culled. Because the listener is moving in a direction away from the reflected sound (b), once it is determined that the reflected sound (b) should be culled, culling of the reflected sound (b) continues automatically thereafter.

[0252] In the above example, the case where the listener moves has been described, but the same can be said for the case where the sound source object 98 moves.

[0253] 32 and 33 will be described. FIGS. 32 and 33 are similar to those in FIG. 17 . As shown in FIG. 32 , in the above-described embodiment, the movement of a listener Δt seconds later is predicted for a listener at a position at time t, and culling of sounds reaching the listener at time t is performed together with culling of sounds reaching a listener at time t + Δt. In contrast, in this example, instead of culling, multiple sounds to be culled are collectively regarded as sounds output from a virtual object 96. This allows multiple sounds to be culled to be replaced with sounds output from a smaller number of virtual objects 96, thereby reducing the amount of calculation. In addition, since multiple sounds to be culled are not completely discarded, degradation of immersive audio due to culling can be avoided to a certain extent.

[0254] FIG. 33 shows a conceptual diagram of the integration process in this example. This diagram illustrates a case where the reverberant sounds (f) and (g) and part of the diffracted sound (h) are selected as sounds to be integrated. As shown in this diagram, the sounds to be integrated by the integration process are output from a small number of virtual objects 96 (virtual sounds of reverberant sounds of (f+g) and (h)). Ｂ ) diffracted sound virtual sound, (h Ａ ) represents the remaining diffracted sounds that were not integrated, and is output from the virtual object 96.

[0255] Here, an example has been described in which a combination of reverberant sounds and a combination of diffracted sounds are each grouped into virtual objects 96, but this is not limiting; sounds with different properties, such as a combination of reverberant sounds and diffracted sounds, may be grouped into a single virtual object 96 and output.

[0256] As a method for combining multiple sounds to be integrated, for example, the sounds to be integrated are added together and considered as a sound output from the virtual object 96. When adding the sounds together, at least one of the energy or phase of each sound may be adjusted before adding them together.

[0257] Although the above description has been given based on the examples, the present disclosure is not limited to only the above examples. For example, the following processing may be performed when the prediction is incorrect.

[0258] If the difference between the estimated value of the position or orientation of the listener or sound source object 98 after Δt and the actual value exceeds a predetermined threshold, it is possible to consider the prediction to be incorrect and perform processing for when the prediction is incorrect. Here, the actual value can be obtained by waiting for the next metadata to be input, allowing for a slight delay, and then obtaining it from the information on the sound source object 98 included in that metadata. As a processing for when the prediction is incorrect, processing using the prediction, i.e., culling processing for sounds that reach the listener in a state at time t + Δt, is stopped.

[0259] Furthermore, although the above explanation has been given using direct sound, reflected sound, reverberant sound, and diffracted sound as the sounds that reach the listener, the above processing may also be applied only to direct sound and reflected sound that have relatively high energy.

[0260] Furthermore, although the contents of the embodiment have been described based on the propagation of sound, the present invention is not limited to the propagation of sound, and similar processing can also be applied to the propagation of light, for example.

[0261] Regarding light propagation, the processing applies to computer graphics that generate scenes based on direct light, reflected light, and diffracted light. Specifically, culling is performed on light that reaches the user at time t using light (direct light, reflected light, diffracted light) that reaches the user from a light source at the user's estimated position at time t + Δt in a virtual space or a space that combines virtual space and real space. This stops the generation of only low-priority light, minimizing the degradation in the quality of the computer graphics presented to the user and significantly reducing the amount of calculation required to generate the computer graphics.

[0262] Other Embodiments Although the embodiments have been described above, the present disclosure is not limited to the above-described embodiments.

[0263] For example, the sound reproduction system described in the above embodiment may be realized as a single device including all of the components, or may be realized by allocating each function to multiple devices and coordinating these multiple devices. In the latter case, the device corresponding to the information processing device may be an information processing device such as a smartphone, tablet terminal, or PC. For example, in the sound reproduction system 100 having a function as a renderer that generates an audio signal with added sound effects, all or part of the renderer's functions may be performed by a server. That is, all or part of the acquisition unit 111, path calculation unit 121, output sound generation unit 131, and signal output unit 141 may be located on a server (not shown). In this case, the sound reproduction system 100 is realized by combining, for example, an information processing device such as a computer or smartphone, a sound presentation device such as a head-mounted display (HMD) or earphones worn by the user 99, and a server (not shown). Note that the computer, sound presentation device, and server may be connected to each other so as to be able to communicate with each other via the same network, or may be connected via different networks. When the sound reproduction system 100 is connected via different networks, the possibility of communication delays increases, so processing on the server may be permitted only when the computer, sound presentation device, and server are connected so as to be able to communicate with each other via the same network. Also, depending on the amount of bitstream data received by the sound reproduction system 100, it may be determined whether the server will take on all or part of the functions of the renderer.

[0264] The sound reproduction system of the present disclosure can also be realized as an information processing device that is connected to a reproduction device having only a driver and that simply reproduces an output sound signal generated based on acquired sound information to the reproduction device. In this case, the information processing device may be realized as hardware having a dedicated circuit, or as software that causes a general-purpose processor to execute specific processing.

[0265] In the above-described embodiment, the processing performed by a specific processing unit may be performed by another processing unit. The order of multiple processing operations may be changed, or multiple processing operations may be performed in parallel.

[0266] In the above-described embodiments, each component may be realized by executing a software program suitable for that component, or by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0267] Furthermore, each component may be realized by hardware. For example, each component may be a circuit (or integrated circuit). These circuits may form a single circuit as a whole, or each may be a separate circuit. Furthermore, each of these circuits may be a general-purpose circuit or a dedicated circuit.

[0268] Furthermore, the general or specific aspects of the present disclosure may be realized as an apparatus, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, etc. Furthermore, the general or specific aspects of the present disclosure may be realized as any combination of an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0269] For example, the present disclosure may be realized as an audio signal reproducing method executed by a computer, or as a program for causing a computer to execute the audio signal reproducing method. The present disclosure may also be realized as a computer-readable non-transitory recording medium on which such a program is recorded.

[0270] In addition, this disclosure also includes forms obtained by applying various modifications to each embodiment that a person skilled in the art would think of, or forms realized by arbitrarily combining the components and functions of each embodiment within the scope that does not deviate from the intent of this disclosure.

[0271] In the present disclosure, the encoded sound information can be rephrased as a bitstream containing a sound signal, which is information about a predetermined sound to be reproduced by the sound reproduction system 100, and metadata, which is information about the localization position when the sound image of the predetermined sound is localized at a predetermined position within a three-dimensional sound field. For example, the sound information may be acquired by the sound reproduction system 100 as a bitstream encoded in a predetermined format such as MPEG-H 3D Audio (ISO / IEC 23008-3). As an example, the encoded sound signal includes information about the predetermined sound to be reproduced by the sound reproduction system 100. The predetermined sound here refers to a sound emitted by a sound source object present in the three-dimensional sound field or a natural environmental sound, and may include, for example, a mechanical sound or the sounds of animals, including humans. When multiple sound source objects are present in the three-dimensional sound field, the sound reproduction system 100 acquires multiple sound signals corresponding to the multiple sound source objects.

[0272] On the other hand, metadata is, for example, information used to control acoustic processing of a sound signal in the sound reproduction system 100. Metadata may be information used to describe a scene expressed in a virtual space (three-dimensional sound field). Here, a scene is a term that refers to a collection of all elements representing three-dimensional images and acoustic events in a virtual space, modeled by the sound reproduction system 100 using metadata. In other words, the metadata here may include not only information for controlling acoustic processing, but also information for controlling video processing. Of course, the metadata may include information for controlling only either audio processing or video processing, or may include information used to control both. In the present disclosure, the bitstream acquired by the sound reproduction system 100 may include such metadata. Alternatively, the sound reproduction system 100 may acquire the metadata separately from the bitstream, as described below.

[0273] The sound reproduction system 100 generates virtual sound effects by performing sound processing on the sound signal using metadata included in the bitstream and additionally acquired position information of the interactive user 99. For example, sound effects such as early reflection sound generation, late reverberation sound generation, diffraction sound generation, distance attenuation effect, localization, sound image localization processing, and Doppler effect may be added. Information for switching on and off all or part of the sound effects may also be added as metadata.

[0274] All or part of the metadata may be obtained from sources other than the bitstream of audio information. For example, either the metadata controlling audio or the metadata controlling video may be obtained from sources other than the bitstream, or both may be obtained from sources other than the bitstream.

[0275] Furthermore, if metadata for controlling the video is included in the bitstream acquired by the audio reproduction system 100, the audio reproduction system 100 may have a function for outputting the metadata that can be used for controlling the video to a display device that displays images or a 3D video reproduction device that reproduces 3D video.

[0276] As an example, the encoded metadata includes information about a three-dimensional sound field including a sound source object emitting a sound and an obstacle object, and information about a localization position when a sound image of the sound is localized at a predetermined position in the three-dimensional sound field (i.e., the sound is perceived as arriving from a predetermined direction), i.e., information about the predetermined direction. Here, an obstacle object is an object that can affect the sound perceived by the user 99 by, for example, blocking or reflecting the sound emitted by the sound source object before it reaches the user 99. Obstacle objects may include not only stationary objects but also animals such as people or moving objects such as machines. Furthermore, when multiple sound source objects exist in a three-dimensional sound field, the other sound source objects may be obstacle objects for any one sound source object. Furthermore, both non-sound-source objects such as building materials or inanimate objects and sound-emitting sound objects may be obstacle objects.

[0277] The spatial information constituting the metadata may include not only the shape of the three-dimensional sound field, but also information representing the shape and position of obstacle objects present in the three-dimensional sound field and the shape and position of sound source objects present in the three-dimensional sound field. The three-dimensional sound field may be either a closed space or an open space, and the metadata may include information representing the reflectance of structures that can reflect sound in the three-dimensional sound field, such as floors, walls, or ceilings, and the reflectance of obstacle objects present in the three-dimensional sound field. Here, the reflectance is the ratio of the energy of reflected sound to incident sound, and is set for each frequency band of sound. Of course, the reflectance may be set uniformly regardless of the frequency band of the sound. Furthermore, when the three-dimensional sound field is an open space, parameters such as a uniform attenuation rate, diffracted sound, or early reflection sound may be used.

[0278] In the above description, reflectance is cited as a parameter related to an obstacle object or a sound source object included in the metadata, but the metadata may include information other than reflectance. For example, information about the material of the object may be included as metadata related to both the sound source object and the non-sound source object. Specifically, the metadata may include parameters such as diffusion rate, transmittance, or sound absorption rate.

[0279] Information about the sound source object may include information such as volume, radiation characteristics (directivity), playback conditions, the number and type of sound sources emitted from a single object, or information specifying a sound source area within the object. The playback conditions may, for example, determine whether the sound is a continuous sound or an event-triggering sound. The sound source area within the object may be determined based on the relative relationship between the position of the user 99 and the position of the object, or may be determined based on the object. When the sound source area is determined based on the relative relationship between the position of the user 99 and the position of the object, the surface from which the user 99 is viewing the object is used as the reference, and the user 99 can be made to perceive sound X as emanating from the right side of the object and sound Y as emanating from the left side of the object as viewed from the user 99. When the sound source area is determined based on the object, the surface from which the user 99 is viewing the object is used as the reference, and the sound emitted from which area of ​​the object can be fixed regardless of the direction the user 99 is viewing. For example, the user 99 can be made to perceive a high-pitched sound coming from the right side and a low-pitched sound coming from the left side when viewing the object from the front. In this case, when the user 99 goes around to the back of the object, the user 99 can be made to perceive a low-pitched sound coming from the right side and a high-pitched sound coming from the left side as viewed from the back.

[0280] The spatial metadata may include the time to early reflections, the reverberation time, or the ratio of direct sound to diffuse sound. If the ratio of direct sound to diffuse sound is zero, the user 99 will perceive only direct sound.

[0281] Furthermore, information indicating the position and orientation of the user 99 in the three-dimensional sound field may be included in the bitstream as metadata as an initial setting, or may not be included in the bitstream. If the information indicating the position and orientation of the user 99 is not included in the bitstream, the information indicating the position and orientation of the user 99 is acquired from information other than the bitstream. For example, the position information of the user 99 in the VR space may be acquired from an app that provides VR content. The position information of the user 99 for presenting sound as AR may be position information obtained by a mobile terminal performing self-position estimation using GPS, a camera, LiDAR (Laser Imaging Detection and Ranging), or the like. Note that the sound signal and metadata may be stored in a single bitstream or may be stored separately in multiple bitstreams. Similarly, the sound signal and metadata may be stored in a single file or may be stored separately in multiple files.

[0282] When an audio signal and metadata are stored separately in multiple bitstreams, information indicating other related bitstreams may be included in one or some of the multiple bitstreams in which the audio signal and metadata are stored. Also, information indicating other related bitstreams may be included in metadata or control information of each of the multiple bitstreams in which the audio signal and metadata are stored. When an audio signal and metadata are stored separately in multiple files, information indicating other related bitstreams or files may be included in one or some of the multiple files in which the audio signal and metadata are stored. Also, information indicating other related bitstreams or files may be included in metadata or control information of each of the multiple bitstreams in which the audio signal and metadata are stored.

[0283] Here, the related bitstreams or files are, for example, bitstreams or files that may be used simultaneously during audio processing. Furthermore, information indicating other related bitstreams may be collectively described in the metadata or control information of one bitstream among multiple bitstreams storing audio signals and metadata, or may be separately described in the metadata or control information of two or more bitstreams among the multiple bitstreams storing audio signals and metadata. Similarly, information indicating other related bitstreams or files may be collectively described in the metadata or control information of one file among multiple files storing audio signals and metadata, or may be separately described in the metadata or control information of two or more files among the multiple files storing audio signals and metadata. Furthermore, a control file collectively describing information indicating other related bitstreams or files may be generated separately from the multiple files storing audio signals and metadata. In this case, the control file does not need to store the audio signal and metadata.

[0284] Here, the information indicating the other related bitstream or file may be, for example, an identifier indicating the other bitstream, a file name indicating the other file, a URL (Uniform Resource Locator), or a URI (Uniform Resource Identifier). In this case, the acquisition unit identifies or acquires the bitstream or file based on the information indicating the other related bitstream or file. Furthermore, the information indicating the other related bitstream may be included in metadata or control information of at least some of the bitstreams among a plurality of bitstreams storing audio signals and metadata, and the information indicating the other related file may be included in metadata or control information of at least some of the files among a plurality of files storing audio signals and metadata. Here, the file including information indicating the related bitstream or file may be, for example, a control file such as a manifest file used for content distribution.

[0285] The present disclosure is useful in reproducing sound, for example, by allowing a user to perceive stereoscopic sound.

[0286] 99 User 100 Sound reproduction system 101 Information processing device 102 Communication module 103 Detector 104 Driver 105 Database 111 Acquisition unit 112 Encoded sound information input unit 113 Decoding processing unit 114 Sensing information input unit 115 Correlation calculation unit 121 Path calculation unit 131 Output sound generation unit 132 Reduction processing unit 133 Culling unit 134 Integration unit 141 Signal output unit 300 3D video reproduction device

Claims

1. An audio processing device comprising: an acquisition unit that acquires audio information including an audio signal and information about the state of a sound source object in a three-dimensional sound field; a relative relationship calculation unit that calculates a first relative relationship which is a relative relationship between the state of the sound source object and a user in the three-dimensional sound field at a predetermined time point, and a second relative relationship which is a relative relationship between the state of the sound source object and the user in the three-dimensional sound field at a subsequent time point after the predetermined time point; and a reduction processing unit that generates an output sound signal that does not include a plurality of sound signals generated when generating an output sound signal from the audio signal included in the acquired sound information by reducing sound signals according to the first relative relationship and the second relative relationship.

2. The sound processing device according to claim 1, wherein the state is a position within the three-dimensional sound field.

3. The sound processing device according to claim 1, wherein the state is a direction in the three-dimensional sound field determined by at least one of an azimuth angle and an elevation angle.

4. The sound processing device of claim 1, wherein the state is a position within the three-dimensional sound field and an orientation within the three-dimensional sound field determined by at least one of an azimuth angle or an elevation angle, and the time interval between the specified point in time and the subsequent point in time is different between the first relative relationship and the second relative relationship when the state is a position within the three-dimensional sound field and the first relative relationship and the second relative relationship when the state is an orientation within the three-dimensional sound field.

5. The sound processing device of claim 1, wherein the state is at least one of a position within the three-dimensional sound field and an orientation within the three-dimensional sound field determined by at least one of an azimuth angle or an elevation angle, and the reduction processing unit reduces sound signals depending on whether a change over time in the relative relationship indicated by the second relative relationship with respect to the first relative relationship indicates a movement in a direction away from the sound source object and the user.

6. The sound processing device according to claim 1, wherein the reduction processing unit reduces sound signals among the plurality of sound signals in accordance with the importance set for each sound for the user and the first relative relationship and the second relative relationship.

7. The sound processing device of claim 1, wherein the reduction processing unit reduces sound signals among the plurality of sound signals in accordance with the importance set for each sound to the user and the first relative relationship and the second relative relationship when a difference between the first relative relationship and the second relative relationship exceeds a predetermined threshold value.

8. The sound processing device according to any one of claims 1 to 7, wherein the reduction processing unit includes a culling unit that reduces the sound signal by discarding the sound signal.

9. The sound processing device according to any one of claims 1 to 7, wherein the reduction processing unit includes an integration unit that reduces the at least two sound signals by discarding the at least two sound signals and supplementing the at least two sound signals with a single virtual sound signal obtained by integrating the at least two sound signals.

10. The sound processing device according to claim 1, wherein the reduction processing unit prohibits reduction of the sound signal according to the second relative relationship when a change in the state of at least one of the sound source object and the user between the specified time point and the subsequent time point satisfies a specified condition.

11. The sound processing device according to claim 10, wherein the predetermined condition is that the change in the state of at least one of the sound source object and the user between the predetermined time point and the subsequent time point is discontinuous.

12. The sound processing device according to claim 10, wherein the predetermined condition is that a rate of change of the state of at least one of the sound source object and the user between the predetermined time point and the subsequent time point exceeds a predetermined rate.

13. The sound processing device according to claim 10, wherein the predetermined condition indicates the occurrence of an obstacle between the sound source object and the user.

14. The sound processing device according to claim 10, wherein the predetermined condition is that a deviation between a calculation result of the state of at least one of the sound source object and the user at the subsequent time point and an actual measured value exceeds a predetermined threshold value.

15. An acoustic processing method executed by a computer, comprising: a step of acquiring sound information including an acoustic signal and information on a state of a sound source object in a three-dimensional sound field; a step of calculating a first relative relationship which is a relative relationship of the state between the sound source object and a user in the three-dimensional sound field at a predetermined time point, and a second relative relationship which is a relative relationship of the state between the sound source object and the user in the three-dimensional sound field at a subsequent time point after the predetermined time point; and a step of generating an output sound signal that does not include a plurality of sound signals generated when generating an output sound signal from the acoustic signal included in the acquired sound information, by reducing sound signals in accordance with the first relative relationship and the second relative relationship.

16. A program for causing the computer to execute the acoustic processing method according to claim 15.

Images