Space audio processing method and device of volume sound source and storage medium

Abstract

The invention provides a spatial audio processing method of a volume sound source, spatial audio processing equipment of the volume sound source and a computer readable storage medium. The spatial audio processing method for the volume sound source comprises the following steps: acquiring first position information, a first geometric feature and a first acoustic feature of the sound source, second position information, a second geometric feature and a second acoustic feature of a shielding object, and third position information of a listener to perform ray tracing; determining the projection position of each tracking light on the spherical surface of the listener and the energy intensity corresponding to the projection position according to the light tracking result; determining a spherical harmonic coefficient of the sound source according to the projection position of each tracking light on the spherical surface of the listener and the corresponding energy intensity, so as to generate a multi-channel filter; and filtering a point sound source signal of the sound source using the multi-channel filter to generate a volumetric sound source signal conforming to the first geometric feature.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of audio rendering technology, and in particular to a spatial audio processing method of a volumetric sound source, a spatial audio processing device of a volumetric sound source, and a computer readable storage medium. BACKGROUND

[0002] The volumetric sound source rendering technology can create a virtual auditory target with a specific shape by generating the sound of a three-dimensional object with a specific shape in space. The common implementation method of the volumetric sound source rendering technology is mainly to sample on a virtual sound source and treat each sampling point as a point sound source for point sound source rendering processing. However, this implementation method based on sampling requires a large number of sampling points, resulting in high power consumption and delay in specific application processes. If the number of sampling points is reduced, the generated sound will sound closer to a point sound source, and comb filtering distortion will also occur. In addition, when implementing the head tracking function of virtual reality application, the position of each sound source in space relative to the current listener orientation needs to be recalculated when the listener rotates the head, which requires frequent updating of the sampling point position, resulting in huge computational overhead.

[0003] In order to overcome the above-mentioned defects of the prior art, the present application provides a spatial audio processing method of a volumetric sound source, a spatial audio processing device of a volumetric sound source, and a computer readable storage medium, which can more accurately model the sound source of any shape to avoid comb filtering distortion, and decouple the computational complexity of generating filters in the spherical harmonic domain and the number of light rays, so as to use a large number of light rays to track complex sound sources with a small amount of computational increment, thereby improving the accuracy of volumetric sound source simulation. SUMMARY

[0004] The following presents a simplified summary of one or more aspects in order to provide a basic understanding of such aspects. This summary is not an extensive overview of all contemplated aspects, and is intended to neither identify key or critical elements of all aspects nor delineate the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description that is presented later.

[0005] In order to overcome the above-mentioned defects of the prior art, the present application provides a spatial audio processing method of a volumetric sound source, a spatial audio processing device of a volumetric sound source, and a computer readable storage medium, which can more accurately model the sound source of any shape to avoid comb filtering distortion, and decouple the computational complexity of generating filters in the spherical harmonic domain and the number of light rays, so as to use a large number of light rays to track complex sound sources with a small amount of computational increment, thereby improving the accuracy of volumetric sound source simulation.

[0006] In particular, the spatial audio processing method of a volumetric sound source according to the first aspect of the present application comprises the following steps: obtaining first position information, first geometric characteristics and first acoustic characteristics of a sound source, second position information, second geometric characteristics and second acoustic characteristics of an occluder, and third position information of a listener for ray tracing; determining, according to a result of the ray tracing, a projection position of each ray on a listener sphere and a corresponding energy intensity thereof; determining spherical harmonic coefficients of the sound source according to the projection position of each ray on the listener sphere and the corresponding energy intensity thereof, to generate a multi-channel filter; and filtering a point sound source signal of the sound source using the multi-channel filter to generate a volumetric sound source signal conforming to the first geometric characteristics.

[0007] Further, in some embodiments of the present application, the first position information, the second position information and the third position information can respectively indicate spatial coordinates of the sound source, the occluder and the listener in a same reference coordinate system. In addition, the first geometric characteristics can indicate a plurality of first three-dimensional meshes of the sound source generated by triangulation, wherein the first three-dimensional meshes contain spatial coordinate information, size information and orientation information of the sound source in the reference coordinate system. In addition, the second geometric characteristics can indicate a plurality of second three-dimensional meshes of the occluder generated by triangulation, wherein the second three-dimensional meshes contain spatial coordinate information, size information and orientation information of the occluder in the reference coordinate system. In addition, the first acoustic characteristics can at least indicate sound wave energy intensity transmitted by the sound source in a plurality of directions of the reference coordinate system. In addition, the second acoustic characteristics can at least indicate sound wave transmission coefficients of the plurality of second three-dimensional meshes of the occluder generated by triangulation.

[0008] Further, in some embodiments of the present application, the step of obtaining first position information, first geometric characteristics and first acoustic characteristics of a sound source, second position information, second geometric characteristics and second acoustic characteristics of an occluder, and third position information of a listener for ray tracing comprises: determining, according to the first position information and the first geometric characteristics of the sound source and the third position information of the listener, a vertical angle range and a horizontal angle range of the sound source relative to the listener; emitting a plurality of the rays from the third position of the listener within the vertical angle range and the horizontal angle range; and determining, according to the second position information and the second geometric characteristics of the occluder and a direction vector r i of each of the rays, a corresponding ray tracing result, wherein the ray tracing result indicates whether each of the rays intersects with the sound source, a distance d i of each intersection point from the listener, and a normal vector n iand whether the intersecting traced light ray crosses the occluder.

[0009] Further, in some embodiments of the present application, the step of determining the vertical angle range and the horizontal angle range of the sound source relative to the listener according to the first position information and the first geometric feature of the sound source, and the third position information of the listener comprises: determining a ball center position vector P S and a radius R S of an circumscribed ball of the sound source according to the first position information and the first geometric feature; determining a listener position vector P L of the listener according to the third position information; determining a distance d of the sound source to the listener according to the ball center position vector P S and the listener position vector P L ; and determining the vertical angle range and the horizontal angle range of the sound source relative to the listener according to the radius R S and the distance d.

[0010] Further, in some embodiments of the present application, after determining the vertical angle range and the horizontal angle range of the sound source relative to the listener, and before emitting a plurality of the traced light rays from the third position where the listener is located, the spatial audio processing method further comprises the following steps: obtaining a spatial resolution threshold of a spherical harmonic of the sound source on the listener's sphere; and in response to the vertical angle range and / or the horizontal angle range being less than the spatial resolution threshold, skipping the step of emitting the traced light rays, and generating a multi-channel filter based on the first position information by using a point sound source modeling method.

[0011] Further, in some embodiments of the present application, the step of determining the corresponding ray tracing result according to the second position information and the second geometric feature of the occluder, and the direction vector r i of each of the traced light rays comprises: obtaining a first orientation information of the listener in the reference coordinate system, and combining the third position information to determine a corresponding first rotation matrix R0; rotating the first position information and the first geometric feature according to the first rotation matrix R0 to determine fourth position information and fourth geometric feature of the sound source in a listener coordinate system; rotating the second position information and the second geometric feature according to the first rotation matrix R0 to determine fifth position information and fifth geometric feature of the occluder in the listener coordinate system; and determining whether each of the traced light rays intersects with the sound source, the distance d i of each intersection point to the listener, and the corresponding ray tracing result according to the fourth position information and the fourth geometric feature of the sound source, the fifth position information and the fifth geometric feature of the occluder, and the direction vector r i of each of the traced light rays., each of the intersection points is on a normal vector n of the sound source surface i , and whether the intersection tracking ray crosses the occluder.

[0012] Further, in some embodiments of the present application, the step of determining the projection position of each tracking ray on the listener sphere and its corresponding energy intensity according to the result of the ray tracing comprises: determining the horizontal angle θ and the elevation angle i of each of the tracking rays on the listener sphere respectively according to the direction vector r of each of the tracking rays; and determining the energy intensity e i of each of the tracking rays on the listener sphere respectively according to the result of the ray tracing, the first acoustic characteristic of the sound source and the second acoustic characteristic of the occluder.

[0013] Further, in some embodiments of the present application, the step of determining the spherical harmonic coefficients of the sound source according to the projection position of each tracking ray on the listener sphere and its corresponding energy intensity comprises: determining the corresponding spherical harmonic coefficients of each of the tracking rays respectively according to the horizontal angle θ, the elevation angle i and the energy intensity e of each of the tracking rays on the listener sphere; and weighting summing the spherical harmonic coefficients i corresponding to each of the tracking rays according to the weight c i to determine the spherical harmonic coefficients of the sound source.

[0014] Further, in some embodiments of the present application, the step of determining the spherical harmonic coefficients of the sound source according to the projection position of each tracking ray on the listener sphere and its corresponding energy intensity comprises: monitoring the orientation of the listener; in response to the change of the orientation of the listener, acquiring second orientation information of the listener in the reference coordinate system and combining the third position information to determine the corresponding second rotation matrix R; and determining the updated spherical harmonic coefficients of the listener after the change of the orientation of the listener according to the spherical harmonic coefficients Y and the first rotation matrix R0 of the listener before the change of the orientation of the listener and the second rotation matrix R.

[0015] Further, in some embodiments of the present application, the step of generating the multi-channel filter comprises: generating an initial multi-channel filter according to the spherical harmonic coefficients Y of the listener before the change of the orientation of the listener; and in response to the change of the orientation of the listener, regenerating an updated multi-channel filter according to the updated spherical harmonic coefficients .

[0016] Furthermore, in some embodiments of the present invention, the step of generating a multi-channel filter includes: decoding coefficients of a virtual loudspeaker group that match the spherical harmonic order of the listener's sphere. And the left ear filter parameters corresponding to the virtual speaker position in the HRTF database. Determine the parameters of the spherical harmonic filter Where, θ i and These represent the horizontal and vertical angles of the virtual loudspeaker, respectively; based on the spherical harmonic coefficients of each spherical harmonic m of each order l of the sound source. and its corresponding spherical harmonic filter parameters Generate a binaural filter h L and h R .

[0017] Furthermore, in some embodiments of the present invention, the step of filtering the point source signal of the sound source using the multi-channel filter to generate a volumetric sound source signal conforming to the first geometric feature includes: determining the distance attenuation coefficient α from the sound source to the listener, and the air absorption coefficient β of the sound source in multiple frequency sub-bands, based on the projection positions of each of the tracking rays on the listener's sphere. i Based on the distance attenuation coefficient α and the air absorption coefficient β of each frequency sub-band. i Determine the comprehensive attenuation coefficient α for each of the aforementioned frequency sub-bands. i Using a cross-filter bank, each of the generated filters is decomposed into subbands, and a corresponding attenuation coefficient 'a' is applied to each of the decomposed subband filters. i The corresponding attenuation coefficient a was applied. i The sub-band filters are added together to obtain an updated multi-channel filter; and the updated multi-channel filter is used to filter the point source signal of the sound source to generate a volume source signal that conforms to the first geometric feature.

[0018] Further, the spatial audio processing device for volumetric sound source according to the second aspect of the present application comprises a memory and a processor. The memory stores computer instructions. The processor is connected to the memory and is configured to: obtain first position information, first geometric characteristics and first acoustic characteristics of a sound source, second position information, second geometric characteristics and second acoustic characteristics of an occluder, and third position information of a listener, to perform ray tracing; determine, according to a result of the ray tracing, a projection position of each traced ray on a listener sphere and a corresponding energy intensity thereof; determine spherical harmonic coefficients of the sound source according to the projection position of each traced ray on the listener sphere and the corresponding energy intensity thereof, to generate a multi-channel filter; and filter a point sound source signal of the sound source using the multi-channel filter, to generate a volumetric sound source signal conforming to the first geometric characteristics.

[0019] Further, in some embodiments of the present application, the processor comprises a filter generation unit and a multi-channel filtering unit. The filter generation unit is configured to: obtain first position information, first geometric characteristics and first acoustic characteristics of a sound source, second position information, second geometric characteristics and second acoustic characteristics of an occluder, and third position information of a listener, to perform ray tracing; determine, according to a result of the ray tracing, a projection position of each traced ray on a listener sphere and a corresponding energy intensity thereof; and determine spherical harmonic coefficients of the sound source according to the projection position of each traced ray on the listener sphere and the corresponding energy intensity thereof, to generate a multi-channel filter. The multi-channel filtering unit is configured to: obtain a point sound source signal of the sound source; and filter the point sound source signal using the multi-channel filter, to generate a volumetric sound source signal conforming to the first geometric characteristics.

[0020] Further, in some embodiments of the present application, the spatial audio processing device for volumetric sound source further comprises a sound source module, a position information module, a characteristic module, an inertial sensor and / or a broadcasting module. The sound source module is configured to provide a point sound source signal collected or stored by the sound source module to the processor. The position information module is configured to provide first position information of the sound source, second position information of the occluder and / or third position information of the listener to the processor. The characteristic module is configured to provide first geometric characteristics and / or first acoustic characteristics of the sound source, and / or second geometric characteristics and / or second acoustic characteristics of the occluder to the processor. The inertial sensor is configured to provide orientation information of the listener to the processor. The broadcasting module is configured to broadcast a volumetric sound source signal conforming to the first geometric characteristics provided by the processor.

[0021] In addition, the computer readable storage medium provided by the third aspect of the present application has computer instructions stored thereon. When the computer instructions are executed by a processor, the method for spatial audio processing of a volumetric sound source provided by the first aspect of the present application is implemented. BRIEF DESCRIPTION OF DRAWINGS

[0022] The above features and advantages of the present application will be better understood by reading the detailed description of the embodiments of the present application in conjunction with the following drawings, in which:

[0023] Figure 1 A structural schematic diagram of a device for spatial audio processing of a volumetric sound source is shown, according to some embodiments of the present application.

[0024] Figure 2 A flowchart of a method for spatial audio processing of a volumetric sound source is shown, according to some embodiments of the present application.

[0025] Figure 3 A schematic diagram of a sound source and a listener position is shown, according to some embodiments of the present application.

[0026] Figure 4 A simplified schematic diagram of a sound source and a listener position is shown, according to some embodiments of the present application.

[0027] Figure 5 A schematic diagram of 0-3 order spherical harmonics is shown, according to some embodiments of the present application.

[0028] Figure 6 A flowchart of sub-band filtering is shown, according to some embodiments of the present application.

[0029] REFERENCE NUMERALS:

[0030] 11 processor

[0031] 111 filter generation unit

[0032] 112 multi-channel filter unit

[0033] 12 sound source module

[0034] 13 position information module

[0035] 14 feature module

[0036] 15 inertial sensor

[0037] 16 playback module

[0038] 31 sound source

[0039] 32 circumscribed sphere of the sound source

[0040] 33 listener DETAILED DESCRIPTION

[0041] The following description will illustrate by way of example the embodiments of the present application. Other advantages and effects of the present application will be readily understood from the description disclosed herein by those skilled in the art. Although the description of the present application will be introduced with reference to preferred embodiments, the features of the present application are not limited to the embodiments. On the contrary, the description of the present application with reference to the embodiments is intended to cover other alternatives or modifications of the present application as can occur to those skilled in the art based on the disclosure. In order to provide a thorough understanding of the present application, many specific details of the description will be included. The present application can be implemented without these details. In addition, in order to avoid obscuring the point of the present application, some specific details will be omitted in the description.

[0042] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connecting" should be understood in a broad sense, for example, can be fixed connection, can be detachable connection, or integral connection; can be mechanical connection, can be electrical connection; can be direct connection, can be indirect connection through intermediate medium, can be internal communication of two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0043] In addition, "up", "down", "left", "right", "top", "bottom", "horizontal", "vertical" used in the following description should be understood as the orientation shown in the paragraph and the related drawings. The relative terms are only used for the convenience of description, and do not mean that the device described should be manufactured or operated in a particular orientation, so it should not be understood as a limitation on the present application.

[0044] It can be understood that although the terms "first", "second", "third" and the like can be used herein to describe various components, regions, layers and / or parts, these components, regions, layers and / or parts should not be limited by these terms, and these terms are only used to distinguish different components, regions, layers and / or parts. Therefore, the first component, region, layer and / or part discussed below can be referred to as the second component, region, layer and / or part without departing from some embodiments of the present application.

[0045] As described above, the common implementation method of the volumetric sound source rendering technology mainly is to sample on the virtual sound source, and to take each sampling point as a point sound source, and to perform the point sound source rendering processing. However, this implementation method based on sampling needs a large number of sampling points, thereby causing high power consumption and delay in the specific application process. If the number of sampling points is reduced, the generated sound will sound closer to the point sound source, and the comb filter distortion problem will also occur. In addition, when implementing the head tracking function of the virtual reality application, when the listener rotates the head, the positions of the sound sources in the space relative to the current listener orientation need to be recalculated, which requires frequent updating of the positions of the sampling points, bringing huge calculation overhead.

[0046] In order to overcome the above-mentioned defects existing in the prior art, the present application provides a spatial audio processing method of a volumetric sound source, a spatial audio processing device of a volumetric sound source and a computer readable storage medium, which can more accurately model the sound source of any shape by using the ray tracing technology, to avoid the occurrence of comb filter distortion, and realize the decoupling of the calculation complexity and the number of rays by generating the filter in the spherical harmonic domain, so as to use a large number of rays to track the complex sound source with a small amount of calculation increment, to improve the accuracy of the volumetric sound source simulation.

[0047] In some non-limiting embodiments, the above-mentioned spatial audio processing method of a volumetric sound source provided by the first aspect of the present application can be implemented via the above-mentioned spatial audio processing device of a volumetric sound source provided by the second aspect of the present application. For details, please refer to Figure 1 , Figure 1 A structural schematic diagram of a spatial audio processing device of a volumetric sound source provided according to some embodiments of the present application is shown.

[0048] In Figure 1 the embodiment shown, the above-mentioned spatial audio processing device of a volumetric sound source provided by the first aspect of the present application includes a memory (not shown) and a processor 11. The memory includes but is not limited to the above-mentioned computer readable storage medium provided by the third aspect of the present application, and has computer instructions stored thereon. The processor 11 is connected to the memory, and is configured to execute the computer instructions stored on the memory, to implement the above-mentioned spatial audio processing method of a volumetric sound source provided by the first aspect of the present application.

[0049] Further, the processor 11 can include a filter generation unit 111 and a multi-channel filtering unit 112. Here, the filter generation unit 111 is used to generate a multi-channel filter. The multi-channel filtering unit 112 is used to filter the point sound source signal using the multi-channel filter, to generate a volumetric sound source signal conforming to the first geometric feature.

[0050] Further, the spatial audio processing device of the volume sound source can further comprise a sound source module 12, a position information module 13, a feature module 14, an inertial sensor (IMU) 15 and / or a broadcasting module 16. Herein, the sound source module 12 is configured to provide the processor 11 with the point sound source signal collected or stored by the sound source module 12. The position information module 13 is configured to provide the processor 11 with the first position information of the sound source, the second position information of the occlusion and / or the third position information of the listener. The feature module 14 is configured to provide the processor 11 with the first geometric feature and / or the first acoustic feature of the sound source, and / or the second geometric feature and / or the second acoustic feature of the occlusion. The inertial sensor 15 is configured to provide the processor 11 with the orientation information of the listener. The broadcasting module 16 is configured to broadcast the volume sound source signal conforming to the first geometric feature provided by the processor 11.

[0051] The working principle of the spatial audio processing device of the volume sound source will be described below in combination with some embodiments of the spatial audio processing method of the volume sound source. Those skilled in the art can understand that the embodiments of the spatial audio processing method are only some non-restrictive embodiments provided by the present application, which are intended to clearly demonstrate the main concept of the present application and provide some specific schemes for facilitating the public to implement, rather than to limit the overall function or overall working mode of the spatial audio processing device. Similarly, the spatial audio processing device is also only a non-restrictive embodiment provided by the present application, which does not limit the execution subject or execution order of each step in the spatial audio processing method.

[0052] For details, please refer to Figure 2 , Figure 2 A flowchart of the spatial audio processing method of the volume sound source according to some embodiments of the present application is shown.

[0053] As Figure 2As shown, in the process of spatial audio rendering of a volumetric sound source, the spatial audio processing device provided in the second aspect of the present invention can first obtain, via the filter generation unit 111 in its processor 11, the first position information, first geometric features, and first acoustic features of the sound source, the second position information, second geometric features, and second acoustic features of the occluder, and the third position information of the listener, for ray tracing. In some non-limiting embodiments, the first position information, second position information, and third position information can respectively indicate the spatial coordinates of the sound source, the occluder, and the listener in the same reference coordinate system. The first geometric feature can indicate multiple first three-dimensional meshes generated by triangulation of the sound source, wherein the first three-dimensional meshes contain the spatial coordinate information, size information, and orientation information of the sound source in the reference coordinate system. The second geometric feature can indicate multiple second three-dimensional meshes generated by triangulation of the occluder, wherein the second three-dimensional meshes contain the spatial coordinate information, size information, and orientation information of the occluder in the reference coordinate system. The first acoustic feature can at least indicate the intensity of sound wave energy transmitted by the sound source in multiple directions to the reference coordinate system. This second acoustic feature can at least indicate the acoustic transmission coefficient of multiple second three-dimensional meshes generated by the triangulation method for the obstruction.

[0054] Please refer to the reference for details. Figures 3-5 , Figure 3 A schematic diagram showing the location of a sound source and a listener according to some embodiments of the present invention is provided. Figure 4 A simplified schematic diagram illustrating the location of a sound source and a listener according to some embodiments of the present invention is shown. Figure 5 A schematic diagram of 0th to 3rd order spherical harmonics provided according to some embodiments of the present invention is shown.

[0055] like Figure 3 and Figure 4 As shown, during ray tracing, the spatial audio processing device can first determine the center position vector P of the circumsphere 32 of the sound source based on the first position information and first geometric features of the sound source (e.g., a river) 31. S and radius R S Then, based on the third position information of listener 33, determine the listener position vector P of listener 33. L Here, the center position vector P of the sphere is... S This is the position vector of the sound source 31 relative to the origin of the reference coordinate system. The listener's position vector P... L The position vector of the listener 33 relative to the origin of the reference coordinate system.

[0056] Subsequently, the spatial audio processing device can, based on the aforementioned sphere center position vector P, S and the listener's position vector P L Determine the distance d from sound source 31 to listener 33:

[0057] d = |P s -P L |

[0058] According to the radius R S and the distance d, the vertical angle range and the horizontal angle range γ of the sound source 31 relative to the listener 33 are determined:

[0059]

[0060] According to the preset maximum order N, the spatial resolution threshold γ min of the volumetric sound source 31 supported thereby is determined:

[0061]

[0062] Here, N is the maximum order of spherical harmonics, which can be selected according to the actual computing capacity of the system. As Figure 5 shown, the larger the maximum order N is, the smaller the minimum expansion angle γ min is, and the higher the precision that the spatial audio processing device can achieve is, and the larger the computing amount of the system is. For example, when the maximum order N = 5, the minimum expansion angle γ min ≈ 20.22 degrees.

[0063] Afterwards, in response to the vertical angle range and / or the horizontal angle range γ determined by the calculation being greater than the spatial resolution threshold γ min , the spatial audio processing device can emit a plurality of tracing rays from the third position of the listener 33 within the vertical angle range and the horizontal angle range γ, and determine the corresponding ray tracing result according to the second position information and the second geometric feature of the occlusion, and the direction vector r i of each tracing ray.

[0064] As Figure 3 shown in detail, in the process of determining the ray tracing result, the spatial audio processing device can first obtain the first orientation information of the listener in the reference coordinate system, and determine the corresponding first rotation matrix R0 in combination with the third position information thereof. Afterwards, the spatial audio processing device can rotate the first position information and the first geometric feature according to the first rotation matrix R0 to determine the fourth position information and the fourth geometric feature of the sound source 31 in the listener coordinate system, and rotate the second position information and the second geometric feature of the occlusion according to the first rotation matrix R0 to determine the fifth position information and the fifth geometric feature thereof in the listener coordinate system:

[0065] v0 = R0v

[0066] Afterwards, the spatial audio processing device can determine at least one ray tracing result for each tracing ray according to the fourth position information and the fourth geometric feature of the sound source 31, the fifth position information and the fifth geometric feature of the occluder, and the direction vector r i of each tracing ray, respectively, including whether the tracing ray intersects the sound source 31, the distance d i from each intersection point to the listener 33, the normal vector n i of each intersection point on the surface of the sound source 31, and whether the intersecting tracing ray penetrates the occluder, so as to accurately model the sound source 31 of any shape and avoid comb filtering distortion.

[0067] Please continue to refer to Figure 2 After obtaining the above ray tracing results, the spatial audio processing device can continue to determine the projection position of each tracing ray on the listener sphere and its corresponding energy intensity e i according to the ray tracing results, and determine the spherical harmonic coefficients of the sound source 31 according to the projection position of each tracing ray on the listener sphere and its corresponding energy intensity, so as to generate a multi-channel filter.

[0068] Specifically, the spatial audio processing device can first determine the horizontal angle θ and the elevation angle of each tracing ray on the listener sphere according to the direction vector r i of each tracing ray, respectively, and determine the energy intensity e i of each tracing ray on the listener sphere according to the ray tracing results, the first acoustic feature of the sound source and the second acoustic feature of the occluder, respectively.

[0069]

[0070] Here, when the intersecting tracing ray does not penetrate the occluder, o i is 1. When the intersecting tracing ray penetrates the occluder, o i is the actual transmission coefficient of the penetration surface (o i <1).

[0071] Afterwards, the spatial audio processing device can determine the corresponding spherical harmonic coefficient and its weight c i according to the horizontal angle θ, the elevation angle and the energy intensity e i of each tracing ray on the listener sphere, respectively.

[0072]

[0073]

[0074]

[0075]

[0076] Here, For the combined Legendre function.

[0077] Then, the spatial audio processing device can adjust the weights based on c. i For each tracking ray, the corresponding spherical harmonic coefficient Weighted summation is used to determine the spherical harmonic coefficients of the sound source.

[0078]

[0079] And based on this, a multi-channel filter is generated.

[0080] Specifically, in the process of generating multi-channel filters, the spatial audio processing device can first determine the decoding coefficients of a virtual loudspeaker group that match the spherical harmonic order of the listener's sphere. And the left ear filter parameters corresponding to the virtual speaker position in the HRTF database. Determine the parameters of the spherical harmonic filter

[0081]

[0082] Then, based on the spherical harmonic coefficients of each spherical harmonic m of each order l of the sound source. and its corresponding spherical harmonic filter parameters Generate a binaural filter h L h R Here, θ i and These represent the horizontal and vertical angles of the virtual speaker, respectively.

[0083] Specifically, spatial audio processing equipment can first determine the spherical harmonic coefficients of each spherical harmonic m of each order l based on each channel n of the sound source. and its corresponding spherical harmonic filter parameters The products are summed to generate the left ear filter h. L :

[0084]

[0085] Then, based on each channel n of the sound source, the spherical harmonic coefficients of each spherical harmonic m of each order l are... and its corresponding spherical harmonic filter parameters And the inverse coefficient σ for distinguishing between the left and right ears m The products are summed to generate the right ear filter h. R :

[0086]

[0087]

[0088] Thus, the present invention can decouple the computational complexity of processing and generating filters in the spherical harmonic domain from the number of rays by generating filters in the spherical harmonic domain, thereby using a large number of rays to track complex sound sources 31 with a small computational increment, thereby improving the simulation accuracy of volumetric sound sources 31.

[0089] After that, such as Figure 2 As shown, the spatial audio processing device can use the generated multi-channel filter h via the multi-channel filtering unit 112 in the processor 11. L h R The point sound source signal of sound source 31 is filtered to generate a volume sound source signal that conforms to its first geometric characteristics.

[0090] Please refer to the details. Figure 6 , Figure 6 A schematic flowchart of a subband filter provided according to some embodiments of the present invention is shown.

[0091] like Figure 6 As shown, during the filtering process of the point source signal of the sound source 31, the multi-channel filtering unit 112 can determine the distance attenuation coefficient α from the sound source 31 to the listener 33, and the air absorption coefficient β of the sound source 31 in multiple frequency sub-bands i, based on the projection position of each tracking ray on the listener's sphere. i :

[0092]

[0093]

[0094] Here, min(d) i To track the minimum distance between each intersection point of the light and the sound source 31 and the listener, β0 is the air absorption constant, with units of dB / m.

[0095] Subsequently, the multi-channel filter unit 112 can adjust the filter based on the distance attenuation coefficient α and the air absorption coefficient β of each frequency sub-band. i Determine the overall attenuation coefficient 'a' for each frequency sub-band. i :

[0096] a i =α*β i .

[0097] Then, the multi-channel filtering unit 112 can use a cross-filter bank to process the generated left ear filter h. L and right ear filter hR Perform subband decomposition and apply the corresponding attenuation coefficient 'a' to each of the resulting subband filters. i Then, the corresponding attenuation coefficient a is applied. i The sub-band filters are summed to obtain an updated multi-channel filter. Finally, the multi-channel filtering unit 112 can use the updated multi-channel filter to filter the point source signal of the sound source to generate a volume source signal that conforms to the first geometric characteristics.

[0098] y L =x*h L

[0099] y R =x*h R

[0100] This adapts to the high-resolution rendering requirements of spatial audio from volumetric sound source 31 for Extended Reality (XR) display devices.

[0101] Furthermore, in some embodiments, in order to simulate in real time the impact of multi-degree-of-freedom (e.g., 6DOF) displacement of the user's (i.e., listener 33) head on the auditory effect of spatial audio emitted by the volumetric sound source 31, the spatial audio processing device can also acquire in real time the first position information, first geometric features and first acoustic features of the sound source 31, the second position information, second geometric features and second acoustic features of the obstruction, and the third position information of the listener 33, so as to perform ray tracing and update the spherical harmonic coefficients of the sound source 31 according to the updated ray tracing results. This allows for the generation of updated multi-channel filters in real time.

[0102] Those skilled in the art will understand that the above-described embodiments of acquiring in real time the first position information, first geometric features and first acoustic features of the sound source 31, the second position information, second geometric features and second acoustic features of the obstruction, and the third position information of the listener 33 to update the multi-channel filter are merely some non-limiting implementations provided by the present invention, intended to clearly demonstrate the main concept of the present invention and provide some specific solutions that are easy for the public to implement, rather than being used to limit the scope of protection of the present invention.

[0103] Optionally, in some other embodiments, the spatial audio processing device can also monitor the orientation of the listener 33 in real time for scenarios where the listener 33 turns around, turns his head, or the like, without obvious change in position. In response to a change in the orientation of the listener 33, the spatial audio processing device can obtain second orientation information of the listener 33 in the reference coordinate system, and combine the third position information thereof to determine a corresponding second rotation matrix R. Then, the spatial audio processing device can directly determine the updated spherical harmonic coefficients Y of the listener 33 after the change in the orientation thereof according to the spherical harmonic coefficients Y before the change in the orientation of the listener 33 and the first rotation matrix R0, and the second rotation matrix R after the change in the orientation of the listener 33.

[0104]

[0105] Then, in the process of generating the multi-channel filter, in response to the change in the orientation of the listener 33, the spatial audio processing device can directly determine the updated multi-channel filter according to the updated spherical harmonic coefficients Y of the listener 33.

[0106] In this way, the present application can realize decoupling of the listener direction and the sound source modeling through ray tracing and spherical projection, to support rotation of the spherical harmonic coefficients after the modeling is completed, so as to realize dynamic tracking of the head pose through very small amount of calculation in scenarios where the position of the listener 33 does not change significantly.

[0107] In addition, in some embodiments of the present application, in order to save the amount of calculation and circumvent the limitation of the spatial resolution of the spherical harmonic, in response to the judgment result that the vertical angle range and / or the horizontal angle range γ is less than the spatial resolution threshold γ min , the spatial audio processing device can also skip the step of emitting the tracing ray, and generate the multi-channel filter based on the first position information by using the point sound source modeling method. Here, the specific implementation of generating the multi-channel filter by using the point sound source modeling method does not involve the technical improvement of the present application, and thus is not described herein.

[0108] In this way, the present application can determine the sound source modeling method to be used according to the visible angle of the sound source 31 on the listener sphere. When the visible angle of the sound source 31 is large enough, the volumetric sound source modeling method is used to generate a high-quality multi-channel filter about the volumetric sound source, and when the visible angle of the sound source 31 is too small, the modeling method of the point sound source is automatically switched to generate a multi-channel filter about the point sound source, to save the amount of calculation of the audio rendering.

[0109] ​In summary, the spatial audio processing method of the volume sound source, the spatial audio processing device of the volume sound source and the computer readable storage medium provided by the present application can more accurately model the sound source of any shape by using the ray tracing technology, so as to avoid the comb filter distortion, and realize the decoupling of the calculation complexity and the number of rays by generating the filter in the spherical harmonic domain, so as to use a large number of rays to track the complex sound source with a small amount of calculation increment, thereby improving the accuracy of the volume sound source simulation.

[0110] Although the above-described methods are illustrated and described as a series of acts, it will be appreciated that the methods are not limited by the order of acts as some acts can, in accordance with one or more embodiments, occur simultaneously or in different orders than shown and described herein. Moreover, not all acts can be required for the practice or implementation of the methods.

[0111] Those skilled in the art will appreciate that information, signals, and data can be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that can be referenced throughout the above description can be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0112] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans can implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

[0113] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to those skilled in the art, and the generic principles defined herein can be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the present disclosure is not intended to be limited to the examples described herein and designs described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method of spatial audio processing of a volumetric sound source, characterized by, The method comprises the following steps: obtaining first position information, first geometric characteristics and first acoustic characteristics of a sound source, second position information, second geometric characteristics and second acoustic characteristics of an occluder, and third position information of a listener for ray tracing; determining, according to a result of the ray tracing, a projection position of each of the traced rays on a listener sphere and a corresponding energy intensity thereof; determining spherical harmonic coefficients of the sound source according to the projection position of each of the traced rays on the listener sphere and the corresponding energy intensity thereof, to generate a multi-channel filter; and filtering a point sound source signal of the sound source using the multi-channel filter to generate a volume sound source signal conforming to the first geometric characteristics. The first position information, the second position information and the third position information respectively indicate spatial coordinates of the sound source, the occluder and the listener in a same reference coordinate system, and / or 2. The spatial audio processing method of claim 1, wherein, The first geometric characteristics indicate a plurality of first three-dimensional meshes of the sound source generated by triangulation, wherein the first three-dimensional meshes contain spatial coordinate information, size information and orientation information of the sound source in the reference coordinate system, and / or The second geometric characteristics indicate a plurality of second three-dimensional meshes of the occluder generated by triangulation, wherein the second three-dimensional meshes contain spatial coordinate information, size information and orientation information of the occluder in the reference coordinate system, and / or The first acoustic characteristics at least indicate sound wave energy intensities transmitted by the sound source in a plurality of directions of the reference coordinate system, and / or The second acoustic characteristics at least indicate sound wave transmission coefficients of the plurality of second three-dimensional meshes of the occluder generated by triangulation. The step of obtaining the first position information, the first geometric characteristics and the first acoustic characteristics of the sound source, the second position information, the second geometric characteristics and the second acoustic characteristics of the occluder, and the third position information of the listener for ray tracing comprises:

3. The spatial audio processing method of claim 1, wherein, determining a vertical angle range and a horizontal angle range of the sound source relative to the listener according to the first position information and the first geometric characteristics of the sound source and the third position information of the listener; emitting a plurality of the traced rays from the third position of the listener within the vertical angle range and the horizontal angle range; and The step of determining the vertical angle range and the horizontal angle range of the sound source relative to the listener according to the first position information and the first geometric characteristics of the sound source and the third position information of the listener comprises: According to the second position information and the second geometric feature of the occlusion object, and the direction vector r of each of the tracking light rays i , determine the corresponding ray tracing result, wherein the ray tracing result indicates whether each of the tracking light rays intersects the sound source, the distance d i of each intersection point from the listener, the normal vector n i of each of the intersection points on the sound source surface, and whether the intersecting tracking light ray passes through the occlusion object.

4. The spatial audio processing method of claim 3, wherein, After determining the vertical angle range and the horizontal angle range of the sound source relative to the listener, and before emitting a plurality of the traced rays from the third position of the listener, the spatial audio processing method further comprises the following steps: According to the first position information and the first geometric feature, a position vector P of a sphere center of a circumscribed sphere of the sound source is determined S and a radius R S ; determining a listener position vector P of the listener based on the third position information L ; determining a distance d of the sound source to the listener based on the sphere center position vector P S and the listener position vector P L ; and According to the radius R S and the distance d, a vertical angle range and a horizontal angle range of the sound source relative to the listener are determined.

5. The spatial audio processing method of claim 4, wherein, obtaining a preset spatial resolution threshold; and in response to the vertical angle range and / or the horizontal angle range being smaller than the spatial resolution threshold, skipping the step of emitting the traced rays and generating a multi-channel filter based on the first position information using a point sound source modeling method. ​ 6. The spatial audio processing method of claim 3, wherein, The second position information and the second geometric feature of the occlusion object, and the direction vector r of each of the tracking rays i The step of determining the corresponding ray tracing result comprises: obtaining first orientation information of the listener in the reference coordinate system, and combining the third position information to determine a corresponding first rotation matrix R0; rotating the first position information and the first geometric feature according to the first rotation matrix R0 to determine fourth position information and fourth geometric feature of the sound source in the listener coordinate system; rotating the second position information and the second geometric feature according to the first rotation matrix R0 to determine fifth position information and fifth geometric feature of the occluder in the listener coordinate system; and According to the fourth position information and the fourth geometric feature of the sound source, the fifth position information and the fifth geometric feature of the barrier, and the direction vector r of each of the tracing rays i , determine whether each of the tracing rays intersects the sound source, the distance d of each intersection point from the listener i , the normal vector n of each of the intersection points on the surface of the sound source i , and whether the intersecting tracing ray passes through the barrier.

7. The spatial audio processing method of claim 3, wherein, the step of determining the projection position of each traced light ray on the listener sphere and the corresponding energy intensity according to the result of the ray tracing comprises: According to a direction vector r of each of the traced light rays i , a horizontal angle θ and an elevation angle φ of each of the traced light rays on the listener sphere are determined, respectively and According to the light ray tracing result, the first acoustic characteristic of the sound source, and the second acoustic characteristic of the barrier, energy intensity e of each of the tracing light rays on the listener sphere is determined respectively i .

8. The spatial audio processing method of claim 7, wherein, the step of determining the spherical harmonic coefficients of the sound source according to the projection position of each traced light ray on the listener sphere and the corresponding energy intensity comprises: According to the horizontal angle θ, the elevation angle φ and the energy intensity e of each of the described tracking light rays on the listener sphere i , respectively determine its corresponding spherical harmonic coefficient with the weight c i ; and​ According to the weight c i , the spherical harmonic coefficients corresponding to each of the traced rays Weighted summation to determine the spherical harmonic coefficients Y of the sound source l m .

9. The spatial audio processing method of claim 1, wherein, the step of determining the spherical harmonic coefficients of the sound source according to the projection position of each traced light ray on the listener sphere and the corresponding energy intensity comprises: monitoring the orientation of the listener; in response to a change in the orientation of the listener, obtaining second orientation information of the listener in the reference coordinate system, and combining the third position information to determine a corresponding second rotation matrix R; and determining updated spherical harmonic coefficients of the listener after changing orientation based on the spherical harmonic coefficients Y of the listener before changing orientation and a first rotation matrix R0 of the listener before changing orientation, and a second rotation matrix R of the listener after changing orientation 10. The spatial audio processing method of claim 9, wherein, the step of generating the multi-channel filter comprises: generating an initial multi-channel filter according to the spherical harmonic coefficients Y before the listener changes the orientation; and in response to a change in orientation of the listener, generating an updated multichannel filter in accordance with the updated spherical harmonic coefficients re-generate an updated multichannel filter.

11. The spatial audio processing method of claim 1, wherein, the step of generating the multi-channel filter comprises: Decoding coefficients of a virtual loudspeaker set matching spherical harmonic orders of a spherical surface of the listener and left ear filter parameters of the corresponding virtual loudspeaker position in the HRTF database Determining spherical harmonic filter parameters wherein θ i and denote horizontal and elevation angles of the virtual loudspeaker, respectively According to the spherical harmonic coefficients Y of each spherical harmonic m of each order l of the sound source l m and its corresponding spherical harmonic filter parameters Generating binaural filters h L and h R .

12. A spatial audio processing method according to claim 11 wherein, the step of filtering the point sound source signal of the sound source using the multi-channel filter to generate the volume sound source signal conforming to the first geometric feature comprises: determining a distance attenuation coefficient a of the sound source to the listener and an air absorption coefficient b of the sound source at a plurality of frequency subbands based on the projected positions of the respective traced rays on the listener sphere i ; According to the distance attenuation coefficient a and the air absorption coefficient β of each of the frequency subbands i , a comprehensive attenuation coefficient a of each of the frequency subbands is determined i ; The generated filters are respectively subjected to subband decomposition using a cross filter bank, and respective subband filters obtained by the decomposition are respectively applied with corresponding attenuation coefficients a i ; adding each of the subband filters to which a corresponding attenuation coefficient a i is applied to obtain an updated multichannel filter; and filtering the point sound source signal of the sound source using the updated multi-channel filter to generate the volume sound source signal conforming to the first geometric feature.

13. A spatial audio processing device of a volumetric sound source, characterized in that, comprise: a memory having computer instructions stored thereon; and a processor connected to the memory and configured to: obtain first position information, first geometric feature and first acoustic feature of a sound source, second position information, second geometric feature and second acoustic feature of an occluder, and third position information of a listener to perform ray tracing; determine the projection position of each traced light ray on the listener sphere and the corresponding energy intensity according to the result of the ray tracing; determine the spherical harmonic coefficients of the sound source according to the projection position of each traced light ray on the listener sphere and the corresponding energy intensity to generate a multi-channel filter; and filter the point sound source signal of the sound source using the multi-channel filter to generate a volume sound source signal conforming to the first geometric feature.

14. The spatial audio processing device of claim 13, wherein, the processor comprises: The filter generating unit is configured to: obtain first position information, first geometric characteristics and first acoustic characteristics of a sound source, second position information, second geometric characteristics and second acoustic characteristics of an occlusion, and third position information of a listener, to perform ray tracing; determine, according to a result of the ray tracing, a projection position of each traced ray on a listener sphere and a corresponding energy intensity thereof; and determine spherical harmonic coefficients of the sound source according to the projection position of each traced ray on the listener sphere and the corresponding energy intensity thereof, to generate a multi-channel filter; and The multi-channel filter unit is configured to: obtain a point sound source signal of the sound source; and filter the point sound source signal using the multi-channel filter, to generate a volumetric sound source signal conforming to the first geometric characteristics.

15. The spatial audio processing device of claim 13, wherein, Further comprising: a sound source module configured to provide a point sound source signal collected or stored by the sound source module to the processor; and / or a position information module configured to provide first position information of the sound source, second position information of the occlusion, and / or third position information of the listener to the processor; and / or a characteristics module configured to provide first geometric characteristics and / or first acoustic characteristics of the sound source, and / or second geometric characteristics and / or second acoustic characteristics of the occlusion to the processor; and / or an inertial sensor configured to provide orientation information of the listener to the processor; and / or a broadcasting module configured to broadcast the volumetric sound source signal conforming to the first geometric characteristics provided by the processor.

16. A computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions, when executed by the processor, implement the spatial audio processing method of a volumetric sound source according to any one of claims 1-12.

Images