Apparatus, method and computer program for synthesizing spatially extended sound sources using elementary spatial sectors - Patents.com
The synthesis of spatially extended sound sources using elementary spatial sectors and frequency-dependent weighting in a lookup table addresses the challenge of realistic rendering in virtual and augmented reality, enhancing audio reproduction efficiency and accuracy.
Patent Information
- Application Number
- JP2024527156
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-11-09
- Filing Date
- 2022-11-07
- Publication Date
- 2024-11-14
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing audio reproduction methods struggle to realistically render spatially extended sound sources, such as musical instruments or ambient sounds, due to frequency attenuation and dependence on listener location, especially in 6DoF virtual and augmented reality scenarios.
A method and apparatus for synthesizing spatially extended sound sources using elementary spatial sectors, incorporating frequency-dependent weighting and occlusion modeling, with precomputed binaural cues stored in a lookup table to efficiently render SESS across various listener positions and orientations.
Enhances the realistic rendering of spatially extended sound sources by efficiently handling occlusion and distance effects, reducing computational load, and maintaining timbre integrity, suitable for 6DoF environments.
Smart Images

Figure 2024542311000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to audio signal processing, and in particular to the synthesis of spatially extended sound sources (SESS). [Background technology]
[0002] The reproduction of sound sources by means of several loudspeakers or headphones has been studied for a long time. The simplest way to reproduce sound sources in such a setting is to render them as point sources, i.e. very (ideally infinitely) small. However, this theoretical concept is unable to model existing physical sound sources in a realistic way. For example, a grand piano has a large vibrating wooden closure with many spatially distributed strings inside, and therefore appears much larger to the ear than a point source (especially when the listener (and microphone) is close to the grand piano). Many real-world sound sources have a significant size ("spatial extent"), such as musical instruments, machines, orchestras or choirs, or ambient sounds (waterfalls).
[0003] The correct / realistic reproduction of such sound sources is the subject of many acoustic reproduction methods, whether binaural using headphones (i.e. using the so-called Head-Related Transfer Function HRTF or Binaural Room Impulse Response BRIR) or more conventionally using loudspeaker setups ranging from two loudspeakers ("stereo") to many loudspeakers arranged in a horizontal plane ("surround sound") and many loudspeakers surrounding the listener in all three dimensions ("3D audio").
[0004] As an example, when listening to a SESS (e.g., a fountain) from a location where part of the fountain is obstructed by a bush, the obstructed part of the fountain undergoes a frequency attenuation process, i.e., is attenuated by a particular frequency response determined by the transmission characteristics of the bush. The ability to render such (partially) obstructed SESS parts is not available in the originally described SESS rendering algorithm.
[0005] Similarly, the present invention can be used to realistically render more distant parts of the SESS at lower levels.
[0006] 2D sound source width This section describes methods for rendering augmented sound sources on a 2D surface facing the listener's viewpoint, for example at a particular azimuth range at 0 degrees elevation (as in traditional stereo / surround sound) or at a particular range of azimuth and elevation angles (as in 3D audio or virtual reality with three degrees of freedom ["3DoF"] of user movement, i.e., head rotation in pitch / yaw / roll axes).
[0007] Increasing the apparent width of an audio object panned between two or more loudspeakers (generating a so-called phantom image or phantom source) can be achieved by decreasing the correlation of the involved channel signals (Blauert, 2001, p. 241-257). As the correlation decreases, the spread of the phantom source increases until it covers the whole range between the loudspeakers for correlation values close to 0 (and the opening angle is not too wide).
[0008] Decorrelated versions of the source signal are obtained by deriving and applying a suitable decorrelation filter. Lauridsen (Lauridsen, 1954) proposed adding / subtracting a time-delayed and scaled version of the source signal to itself to obtain two decorrelated versions of the signal. More complex approaches were proposed, for example, by Kendall (Kendall, 1995), who iteratively derived pair-decorrelation all-pass filters based on combinations of random number sequences. Faller et al. proposed a suitable decorrelation filter ("diffuser") in (Baumgarte & Faller, 2003). Zotter et al. also derived a filter pair in which frequency-dependent phase or amplitude differences are used to achieve the broadening of phantom sources (Zotter & Frank, 2013). Furthermore, (Alary, Politis, & Valimaki, 2017) proposed a decorrelation filter based on velvet noise, which was further optimized by (Schlecht, Alary, Valimaki, & Habets, 2018).
[0009] In addition to reducing the correlation of the corresponding channel signals of phantom sources, the source width can also be increased by increasing the number of phantom sources due to audio objects. In (Pulkki, 1999), the source width is controlled by panning the same source signal in (slightly) different directions. This method was first proposed to stabilize the perceived phantom source spread of VBAP-panned (Pulkki, 1997) source signals when these source signals are moved in the acoustic scene. This is advantageous because, depending on the source direction, the rendered source may be reproduced by two or more loudspeakers, resulting in an undesirable modification of the perceived source width.
[0010] Virtual World DirAC (Pulkki, Laitinen, & Erkut, 2009) is an extension of the traditional Directional Audio Coding (DirAC) (Pulkki, 2007) approach for speech synthesis in virtual worlds. To render spatial coverage, directional sound components of a sound source are panned randomly within a certain range around the original direction of the sound source, and the panning direction varies with time and frequency.
[0011] A similar approach was pursued in (Pihlajamaki, Santala, & Pulkki, 2014), where spatial coverage is achieved by randomly distributing the frequency bands of the source signal into different spatial directions, a method that aims to generate a spatially distributed, enveloping sound that arrives equally from all directions, rather than controlling the exact degree of coverage.
[0012] Verron et al. achieved spatial coverage of the source without using panned correlated signals by synthesizing multiple incoherent versions of the source signal, distributing them evenly on a circle around the listener, and mixing between them (Verron, Aramaki, Kronland-Martinet, & Pallone, 2010). The number and gain of simultaneously active sources determines the strength of the broadening effect. This method was implemented as a spatial extension to an ambient sound synthesizer.
[0013] 3D sound source width This section describes methods related to rendering augmented sound sources in a stereoscopic manner as required for virtual reality in 3D space, i.e. six degrees of freedom ("6DoF"), meaning six degrees of freedom of movement of the user, i.e. head rotation in pitch / yaw / roll axes + three translational directions x / y / z.
[0014] Potard et al. extended the concept of source extent as a one-dimensional parameter of a sound source (i.e. its width between two loudspeakers) by studying the perception of source shape (Potard, 2003). They generated multiple incoherent point sources by applying a (time-varying) decorrelation technique to the original source signal and then placing the incoherent sources at different spatial locations, thereby giving them a three-dimensional extent (Potard & Burnett, 2004).
[0015] In MPEG-4 Advanced AudioBIFS (Schmidt & Schroder, 2004), volumetric objects / shapes (skins, boxes, ellipsoids, and cylinders) can be filled with a number of equally distributed, decorrelated sound sources to evoke a three-dimensional source range.
[0016] To extend and control the source range using Ambisonics, Schmele et al. (Schmele & Sayin, 2018) proposed a combination of reducing the Ambisonics order of the input signal, which essentially increases the apparent source width, and distributing decorrelated copies of the source signal around the listening space.
[0017] Another approach was introduced by Zotter et al., who adopted the principles proposed for Ambisonics in (Zotter & Frank, 2013) (i.e., deriving filter pairs that introduce frequency-dependent phase and magnitude differences to achieve source range in a stereo reproduction setup) (Zotter F., Frank, Kronlachner, & Choi, 2014).
[0018] A common drawback of panning-based approaches (e.g., (Pulkki, 1997) (Pulkki, 1999) (Pulkki, 2007) (Pulkki, Laitinen, & Erkut, 2009)) is their dependency on the listener's position. Even a slight deviation from the sweet spot causes the spatial image to be contracted to the loudspeaker closest to the listener. This significantly limits their application in six degrees of freedom (6DoF) virtual and augmented reality contexts, where the listener is assumed to be free to move around. Furthermore, the distribution of time-frequency bins in DirAC-based approaches (e.g., (Pulkki, 2007) (Pulkki, Laitinen, & Erkut, 2009)) does not always guarantee a proper rendering of the spatial extent of the phantom source. Moreover, this usually results in a significant degradation of the timbre of the source signal.
[0019] Decorrelation of source signals is typically achieved by one of the following methods: i) deriving filter pairs with complementary magnitudes (e.g., (Lauridsen, 1954)), ii) using all-pass filters with constant magnitude but (randomly) scrambled phase (e.g., (Kendall, 1995) (Potard & Burnett, 2004)), or iii) spatially randomly distributing the time-frequency bins of the source signals (e.g., (Pihlajamaki, Santala, & Pulkki, 2014)).
[0020] All approaches entail their own implications: Complementary filtering of source signals according to i) typically results in a change in the perceived timbre of decorrelated signals. All-pass filtering as in ii) preserves the timbre of the source signals, but the scrambled phase disrupts the original phase relationships, causing severe temporal dispersion and smearing artifacts, especially for transient signals. Spatially distributed time-frequency bins have proven effective for some signals, but also change the perceived timbre of the signal. Moreover, this has been shown to be highly signal-dependent, introducing severe artifacts for non-continuous signals.
[0021] Arranging a volumetric shape with multiple decorrelated versions of the source signal, as proposed in Advanced AudioBIFS ((Schmidt & Schroder, 2004) (Potard, 2003) (Potard & Burnett, 2004)), presupposes the availability of a large number of filters that generate mutually decorrelated output signals (typically more than 10 point sources per volumetric shape are used). However, finding such filters is not a trivial task and becomes more difficult the more such filters are needed. Furthermore, if the source signals are not perfectly decorrelated and the listener moves around such a shape, e.g. in a (virtual reality) scenario, the individual source distances to the listener correspond to different delays of the source signal and their superposition at the listener's ears results in position-dependent comb filtering and potentially introduces unpleasant and unstable coloration of the source signal.
[0022] By lowering the Ambisonics order, controlling source width with Ambisonics-based techniques has been shown in (Schmele & Sayin, 2018) to have audio effects only for transitions from 2nd to 1st or 0th order. Furthermore, these transitions are not only perceived as an increase in source width, but are often also perceived as the movement of phantom sources. Adding decorrelated versions of the source signal can help stabilize the perception of apparent source width, but also introduces comb filtering effects that change the timbre of the phantom sources.
[0023] An efficient method for binaural rendering of spatially extended sound sources (SESS) is disclosed in WO 2021 / 180935 using two decorrelated versions of the input waveform signal (which may be generated by generating a decorrelated version of this mono signal using the original mono signal and a decorrelator), a cue calculation stage that calculates the target binaural (and tonal) cues of the spatially extended sound source depending on the size of the sound source (for example given as an azimuth-elevation range depending on the position and orientation of the spatially extended sound source and the listener). In a preferred embodiment, this cue calculation stage pre-calculates target cues depending on the spatial region covered by the SESS and stores them in a look-up table, and a binaural cue adjustment stage that uses the target cues to generate a binaurally rendered output signal from the input signal and its decorrelated version forms the cue calculation stage (look-up table). The binaural adjustment stage adjusts the binaural cues of the input signal (inter-channel coherence ICC, inter-channel phase difference ICPD, inter-channel level difference ICLD) in several steps to their desired target values, as calculated by the cue calculation stage / lookup table. [Prior art documents] [Patent documents]
[0024] [Patent Document 1] International Publication No. 2021 / 180935 Brochure [Non-patent literature]
[0025] [Non-Patent Document 1] Blauert, 2001, S. 241-257 [Non-Patent Document 2] Lauridsen, 1954 [Non-Patent Document 3] Kendall, 1995 [Non-Patent Document 4] Baumgarte&Faller, 2003 [Non-Patent Document 5] Faller&Baumgarte, 2003 [Non-Patent Document 6] Zotter&Frank, 2013 [Non-Patent Document 7] Alary, Politis, & Valimaki, 2017 [Non-Patent Document 8] Schlecht, Alary, Valimaki, & Habets, 2018 [Non-Patent Document 9] Pulkki, 1999 [Non-Patent Document 10] Pulkki, 1997 [Non-Patent Document 11] Pulkki, Laitinen, & Erkut, 2009 [Non-Patent Document 12] Pulkki, 2007 [Non-Patent Document 13] Pihlajamaki, Santala, & Pulkki, 2014 [Non-Patent Document 14] Verron, Aramaki, Kronland-Martinet, & Pallone, 2010 [Non-Patent Document 15] Potard, 2003 [Non-Patent Document 16] Potard&Burnett, 2004 [Non-Patent Document 17] Schmidt&Schroder, 2004 [Non-Patent Document 18] Schmele&Sayin, 2018 [Non-Patent Document 19] Zotter F., Frank, Kronlachner, & Choi, 2014 [Non-Patent Document 20] Lauridsen, 1954 [Summary of the Invention]
[0026] It is an object of the present invention to provide an improved concept for spatially extended sound sources. This object is achieved by the subject matter defined in the independent claims, preferred embodiments being defined in the dependent claims.
[0027] A typical fast synthesis algorithm of spatially extended sound sources (SESS) simulates the sound impression of a diffuse field in a certain specified target spatial region. This is achieved by the (virtual) summation of many closely located sound sources driven by uncorrelated versions of the audio signal. Sometimes, parts of the SESS are occluded by partially transparent materials (e.g. bushes), resulting in frequency-selective attenuation of the SESS in the occluded spatial region. This effect can be elegantly and efficiently incorporated into an efficient SESS algorithm by introducing a weighting step in the calculation between the table lookup operation and the further calculation of the desired binaural cues. The lookup table stores pre-computed partial sums of terms for each spatial sector around the listener. This expansion does not involve substantially additional computational cost. The embodiments relate to an apparatus and a method or a computer program for recreating or synthesizing a spatially extended sound source (SESS) with selective spatial weighting.
[0028] An advantage of the present invention is that it allows the processing of spatially extended sound sources, possibly with complex geometric shapes.
[0029] A further advantage of the present invention is that embodiments allow an improved concept of reproducing spatially extended sound sources, allowing the possibility of spatially selective modification of the SESS rendering.
[0030] The first aspect concerns the use of basic spatial sectors. This first aspect concerns the storage of the data of the basic spatial sectors in a look-up table, the basic spatial sectors being distributed over a sphere. The data of the basic spatial sectors is preferably linked to the user's head forming a user-centric audio scene, and is the same for each tilt of the head in the same position and for each position of the listener's head, i.e., for each degree of freedom of 6-DOF. However, each movement or tilt of the head leads to a situation where the sound from the SESS "enters" the user's head in another basic spatial sector or sectors. The renderer determines the basic spatial sectors covered by the SESS, retrieves the stored data for these specific sectors, performs weighting of the stored data due to occluding objects or specific distances if necessary, then combines the stored data (or the weighted stored data, in case of weighting), and then uses the result of the combining operation for rendering (e.g., the rendering cues are calculated from the combined (co)variance data, but other steps and parameters can also be used here. Thus, this aspect may or may not use reference to occluding objects, and may or may not use reference to specific stored variance data, since the combining (and optionally also the weighting) can also be done if other data is stored, such as the (average) HRTFs (for the basic spatial sectors or for the entire spatial range) or the frequency-dependent cues themselves.
[0031] The second aspect relates to modifying objects, which may be occluding objects or other objects, so that the sound of the SESS is modified on the way from the SESS position to the user with a certain position and / or inclination. This second aspect relates to the processing of occluding objects, for example. The effect of the occluding object is a frequency-dependent attenuation with low-pass characteristics. The frequency-dependent weighting can also be applied to the prior art procedures when they do not have basic spatial sectors. Based on the transmitted data describing the occluding object, the SESS must decide whether it is occluded or not, and then apply the occlusion function to the stored cues, for example frequency-dependent, already given for different frequencies in the prior art. This is therefore a useful application of the occlusion effect in the prior art, without using basic spatial sectors or without using stored distribution data.
[0032] A third aspect relates to the storage of variance and covariance data of HRTFs, for example, for different spatial ranges or basic spatial sectors. This third aspect relates to the storage of variance and covariance data of HRTFs, for example, in a storage location, for example, in a look-up table. It is not important whether this data is stored for a certain spatial range, as in the prior art, or for basic spatial sectors. The renderer then calculates all rendering cues on the fly from the stored variance data. In contrast to the prior art applications, where at least the IACC is stored and possibly other cues or HRFT data, this is not done in this aspect. The covariance data is stored and the cues are calculated on the fly. This aspect may or may not use basic spatial sectors, and may or may not use corrections or occluding objects.
[0033] All aspects can be used separately from each other, together with each other, or only any two selected aspects can be combined.
[0034] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. [Brief description of the drawings]
[0035] [Figure 1] FIG. 1 illustrates an apparatus for synthesizing a spatially extended sound source according to a first embodiment of the present invention; [Figure 2a] FIG. 2 shows an apparatus for synthesizing a spatially extended sound source according to a second embodiment of the present invention; [Figure 2b] FIG. 2 illustrates an audio scene generator according to a second embodiment of the present invention. [Diagram 3] FIG. 2 shows a preferred embodiment of the third aspect of the present invention. [Figure 4] FIG. 2 is a block diagram for explaining certain parts of an embodiment of the present invention. [Diagram 5] FIG. 2 is another block diagram for explaining some portions of aspects of the present invention. [Figure 6] FIG. 2 is a further block diagram for illustrating some aspects of the present invention. [Figure 7] 4 illustrates an exemplary division of the rendering range in a basic spatial sector. [Figure 8] FIG. 1 illustrates a procedure for combining three aspects of the invention for the synthesis of spatially extended sound sources. [Figure 9] FIG. 7 illustrates a preferred embodiment of block 320 of FIGS. [Figure 10] FIG. 2 illustrates an embodiment of a second channel processor. [Figure 11] FIG. 1 is a schematic diagram particularly illustrating features of the first and second aspects of the present invention. [Figure 12] FIG. 1 is a diagram for explaining the first, second, and third aspects of the invention. [Figure 13] FIG. 11 illustrates the decorrelator of FIG. 10 connected to an audio processor synthesis according to a further embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0036] Fig. 1 shows an apparatus for synthesizing a spatially extended sound source. The apparatus comprises a storage device 2000 for storing rendering data items for different basic spatial sectors covering a rendering range for a listener. The apparatus further comprises a sector identification processor 4000 for identifying a set of basic spatial sectors belonging to a particular spatially extended sound source from the different basic spatial sectors. The identification is performed based on listener data and data related to the spatially extended sound source (SESS). The apparatus further comprises a target data calculator 5000 for calculating target rendering data from the rendering data items for the set of basic spatial sectors. The apparatus further comprises an audio processor 3000 for processing an audio signal representing the spatially extended sound source using the target rendering data generated by the target data calculator 5000.
[0037] 2a shows an apparatus for synthesizing a spatially extended sound source (SESS) comprising an input interface 4020 for receiving an audio scene description, the audio scene description comprising spatially extended sound source data for a spatially extended sound source and correction data for potential correction objects. Further, the input interface 4020 is configured to receive listener data.
[0038] The sector identification processor 4000, which may generally be implemented as the sector identification processor 4000 of Fig. 1, is configured to identify a restricted modified spatial sector of the spatially extended sound source within a rendering range of a listener, the rendering range of the listener being larger than the restricted modified spatial sector. The identification is performed based on the spatially extended sound source data, the listener data and the modified data. Furthermore, the device generally comprises a target data calculator 5000, which may generally be implemented the same as or similar to the target data calculator 5000 of Fig. 1. The device is configured to calculate target rendering data from one or more rendering data items belonging to the modified restricted spatial sector as determined by the block 4000 of Fig. 2a. Furthermore, the device for synthesizing a spatially extended sound source according to the second embodiment shown in Fig. 2a comprises an audio processor for processing an audio signal representing the spatially extended sound source using the modified data, i.e. the target rendering data affected by data of modified objects such as occluding objects.
[0039] Fig. 2b shows an audio scene generator, also according to the second embodiment, comprising a spatially extended sound source data generator 6010, a correction data generator 6020 and an output interface 6030. The spatially extended sound source data generator 6010 is configured to generate data of the spatially extended sound source and provide this data to the output interface. This data preferably includes at least one of position information, direction information and geometry data for the spatially extended sound source as metadata for the spatially extended sound source, and in addition waveform data for the SESS, such as a stereo signal for the SESS in case of a large SESS such as a grand piano, or only a mono signal for the SESS data processed by a decorrelation device as shown in element 310 of Fig. 10 or element 3100 of Fig. 13.
[0040] The correction data generator 6020 is configured to generate the correction data, which may include a description of a low pass function or a description of geometry data for the potential correction object. In one embodiment, the low pass function includes attenuation values for higher frequencies, which represent stronger attenuation values compared to the attenuation values for lower frequencies, and this data is forwarded to the output interface 6030 for insertion into the generated audio scene description.
[0041] The audio scene description shown in Figure 2b is therefore extended compared to the SESS description in that it not only contains SESS data but also data on modification objects, which are elements that are not sound sources themselves but which modify the sound field generated by the sound sources.
[0042] FIG. 3 shows a preferred embodiment of an apparatus for synthesizing a spatially extended sound source according to the third aspect.
[0043] The element comprises a storage device for storing one or more rendering data items for different restricted spatial sectors, the different restricted spatial sectors being located within a rendering range of a listener, and the one or more rendering data items for the restricted spatial sectors comprising at least one of a left variance data item, a right variance data item, and a left / right covariance data item.
[0044] Further, the apparatus comprises a sector identification processor 4000 for identifying one or more restricted spatial sectors for spatially extended sound sources within the rendering range of the listener based on the spatially extended sound source data, preferably based on the listener's position or orientation.
[0045] The left variance data, right variance data and covariance data are input to a target data calculator 5000 for calculating target rendering data from the stored left variance data, the stored right variance data or the stored covariance data corresponding to one or more restricted spatial sectors as determined by the sector identification processor 4000. The target rendering data is forwarded to an audio processor 3000 for processing an audio signal representing a spatially extended sound source using the target rendering data. In general, the audio processor 3000 can be implemented in the same way as in Figures 1 and 2b or Figures 4, 5 and 6, or the audio processor 3000 can be implemented in a different way.
[0046] Preferably, the left variance data item, the right variance data item and / or the left / right covariance data item are data items related to head-related transfer function data or related to binaural room impulse response data or related to binaural room transfer function data or related to head-related impulse response data. Furthermore, the rendering data item comprises variance or covariance data item values for different frequencies such that frequency selective / frequency dependent processing is achieved.
[0047] In particular, the storage device 2000 is configured to store, for each restricted spatial sector, a frequency-dependent representation of a left variance data item, a frequency-dependent representation of a right variance data item, and a frequency-dependent representation of a covariance data item.
[0048] Upstream processing of stored variance / covariance data items is illustrated in several figures of WO 2021 / 180935, shown below as Figures 4, 5, and 6.
[0049] Figure 4 shows a block diagram of the SESS combination, Figure 5 shows another block diagram of the SESS combination simplified according to option 1, and Figure 6 shows a block diagram of the SESS combination simplified according to option 2.
[0050] Fig. 4 shows an embodiment of an apparatus for synthesizing a spatially extended sound source. The apparatus comprises a spatial information interface for receiving a spatial range indication information input indicating a limited spatial range of the spatially extended sound source within a maximum spatial range. The limited spatial range is input to a cue information provider 200 configured to provide one or more cue information items in response to the limited spatial range given by the spatial information interface. The cue information item or several cue information items are provided to an audio processor 300 configured to process an audio signal representing the spatially extended sound source using the one or more cue information items provided by the cue information provider 200. The audio signal of the spatially extended sound source (SESS) may be a single channel, or a first audio channel and a second audio channel, or three or more audio channels. However, in order to lower the processing load, a small number of channels of the spatially extended sound source or the audio signal representing the spatially extended sound source is preferred.
[0051] The audio signal is input to the audio processor 300, which processes the input audio signal or, if the number of input audio channels is less than the required number, e.g. only one, the audio processor comprises, for example, a second channel processor 310 as shown in FIG. 10, which comprises a decorrelator for generating a second audio channel S2 decorrelated from the first audio channel S, also shown in FIG. 10 as S1. The cue information items may be actual cue items such as inter-channel correlation items, inter-channel phase difference items, inter-channel level differences and gain items, e.g. gain coefficient items G1, G2, both representing inter-channel level differences and / or absolute amplitude or power or energy levels, or the cue information items may be actual filter functions such as head-related transfer functions with a number required by the actual number of output channels to be combined in the combined signal. Thus, if the combined signal has two channels, such as two binaural channels or two loudspeaker channels, one head-related transfer function is required for each channel. Instead of head-related transfer functions, head-based impulse response functions (HRIRs) or binaural or non-binaural room impulse response functions (B)RIRs are needed. One such transfer function is needed for each channel, and Figure 4 shows an embodiment with two channels.
[0052] In one embodiment, the cue information provider 200 is configured to provide the inter-channel correlation value as a cue information item. The audio processor 300 is configured to actually receive the first audio channel and the second audio channel via the audio signal interface 305. However, if the audio signal interface 305 receives only a single channel, an optionally provided second channel processor generates the second audio channel, for example by the procedure of Fig. 9. The audio processor performs a correlation process to impose a correlation between the first audio channel and the second audio channel using the inter-channel correlation value.
[0053] Additionally or alternatively, further cue information items can be provided, such as an inter-channel phase difference item, an inter-channel time difference item, an inter-channel level difference and gain item, or a first gain factor and a second gain factor information item. The items may be inter-aural (IACC) correlation values, i.e. more specific inter-channel correlation values, or inter-aural phase difference items (IAPD), i.e. more specific inter-channel phase difference values.
[0054] In a preferred embodiment, correlation is imposed at 320 by the audio processor 300 in response to correlation cue information items before any ICPD (330), ICTD or ICLD (340) adjustments are performed, or before any HRTF or other transfer filter function processing (350) is performed. However, in some cases the order can be set differently.
[0055] In a preferred embodiment, the device comprises a memory for storing information on different cue information items associated with different spatial range notifications. In this situation, the cue information provider further comprises an output interface for retrieving from the memory one or more cue information items associated with the spatial range notifications entered in the corresponding memory. Such a look-up table 210 is for example shown in Fig. 4, Fig. 5 or Fig. 6, where the look-up table comprises a memory and an output interface for outputting the corresponding cue information items. In particular, the memory may be a IACC, IAPD or G as shown in Fig. 1b. l and G r In addition to storing values, the memory in the lookup table can also store filter functions as shown in block 220 of Figures 5 and 6, labeled "Select HRTF". In this embodiment, although shown separately in Figures 5 and 6, blocks 210, 220 may comprise the same memory and store corresponding cue information items such as IACC and optionally IAPD, in conjunction with corresponding spatial range indications, depicted as azimuth and elevation angles, as well as the HRTF for the left output channel. l and the HRTF of the right output channel rThe transfer functions of the filters are stored, and the left and right output channels are S in FIG. 4, FIG. 5, or FIG. 6. l and S r is shown as:
[0056] The memory used by the lookup table 210 or the selection function block 220 may also use a storage device in which corresponding parameters are available based on a particular sector code or sector angle or sector angle range. Alternatively, the memory may store a vector codebook, or a multidimensional function fitting routine, or a Gaussian Mixture Model (GMM) or a Support Vector Machine (SVM), as the case may be.
[0057] The target cue is calculated as follows: Figure 4 shows a general block diagram of the concept. TIFF2024542311000002.tif716 shows the desired source range in terms of azimuth range. TIFF2024542311000003.tif715 is the desired source range in terms of elevation angle range. TIFF2024542311000004.tif712 and TIFF2024542311000005.tif712 shows two uncorrelated input signals, TIFF2024542311000006.tif74 shows the frequency index. Therefore, TIFF2024542311000007.tif712 and For TIFF2024542311000008.tif712, the following formula holds: JPEG2024542311000009.jpg1682(1)
[0058] In addition, both input signals must have the same power spectral density. Alternatively, one input signal It is possible to provide only TIFF2024542311000010.tif710. The second input signal is generated internally using a decorrelation filter as shown in FIG. TIFF2024542311000011.tif712 and Given TIFF2024542311000012.tif712, an augmented sound source is synthesized by successively adjusting the inter-channel coherence (ICC), inter-channel phase difference (ICPD), and inter-channel level difference (ICLD) to match the corresponding inter-aural cues. The quantities required for these processing steps are read from pre-computed look-up tables. The resulting left and right channel signals TIFF2024542311000013.tif712 and TIFF2024542311000014.tif712 can be played through headphones, mimicking SESS. Note that ICPD and ICLD adjustment blocks can be interchanged, although ICC adjustment must be performed first. Instead of IAPD, the corresponding interaural time difference (IATD) can be reproduced as well. However, in the following, only IAPD is considered further.
[0059] In the ICC adjustment block, the cross-correlation between both input signals is adjusted to the desired value |IACC(ω)| using the following equation
[21] : TIFF2024542311000015.tif778(2) TIFF2024542311000016.tif778(3) TIFF2024542311000017.tif1380(4) TIFF2024542311000018.tif1343(5)
[0060] Applying these formulas, the input signal TIFF2024542311000019.tif712 and As long as TIFF2024542311000020.tif712 are completely uncorrelated, the desired cross-correlation is obtained. Furthermore, their power spectral densities must be identical. The corresponding block diagram is shown in Figure 9. Four filters 321 to 324 and two adders 325, 326 process the inputs to obtain the output of block 320. The ICPD adjustment block 330 is described by the following equation: JPEG2024542311000021.jpg1381(6) JPEG2024542311000022.jpg1652(7)
[0061] Finally, ICLD adjustment 340 is performed as follows. JPEG2024542311000023.jpg1164(8) JPEG2024542311000024.jpg1369(9) During the ceremony, TIFF2024542311000025.tif712 represents the left ear gain, TIFF2024542311000026.tif713 represents the right ear gain. This means that TIFF2024542311000027.tif712 and As long as TIFF2024542311000028.tif712 have the same power spectral density, the desired ICLD is obtained. Since the left and right ear gains are used directly, mono spectral cues are reproduced in addition to the IALD.
[0062] To further simplify the aforementioned method, two simplification options are described. As mentioned above, the main interaural cue affecting the perceived spatial range (in the horizontal plane) is the IACC. It is therefore conceivable to not use pre-calculated IAPD and / or IALD values, but to adjust them directly via the HRTF. For this purpose, an HRTF is used that corresponds to a position that represents the desired sound source range. As this position, without loss of generality, the average of the desired azimuth / elevation range is chosen here. In the following, both options are described.
[0063] The first option involves using pre-calculated IACC and IAPD values, however the ICLD is adjusted using the HRTF corresponding to the center of the sound source range.
[0064] A block diagram of the first option is shown in Figure 5, where: TIFF2024542311000029.tif712 and TIFF2024542311000030.tif712 is calculated using the following formula: TIFF2024542311000031.tif644(10) JPEG2024542311000032.jpg13104(11) TIFF2024542311000033.tif735 and TIFF2024542311000034.tif734 describes the location of the HRTF that represents the average of the desired azimuth / elevation range. The main advantages of the first option include:
[0065] No spectral shaping / coloration when the source range is increased compared to a point source in the middle of the source range.
[0066] · TIFF2024542311000035.tif712 and Lower memory requirements compared to a full blown one, since TIFF2024542311000036.tif712 does not need to be stored in a lookup table.
[0067] Since only the resulting ICC and ICPD, and not the ICLD, depend on the HRTF dataset used during pre-computation, it is more robust to changes in the HRTF dataset at run-time compared to the full-blown method.
[0068] The main drawback of this simplified version, compared to a non-extended source, is that it fails whenever a dramatic change in the IALD occurs, i.e. the IALD is not reproduced with sufficient accuracy, for example when the source is not centered at 0° azimuth and at the same time the horizontal source range becomes too large.
[0069] The second option involves using only the pre-calculated IACC value: ICPD and ICLD are adjusted using the HRTF corresponding to the center of the sound source range.
[0070] A block diagram of the second option is shown in Figure 6, where: TIFF2024542311000037.tif712 and TIFF2024542311000038.tif712 is calculated using the following formula: JPEG2024542311000039.jpg1182(12) JPEG2024542311000040.jpg985(13) In contrast to the first option, here the phase and magnitude of the HRTFs are used, rather than just the magnitude, which allows adjusting not only the ICLD but also the ICPD.
[0071] First, the (co)variance terms between the left and right channels are calculated as follows: TIFF2024542311000041.tif129116
[0072] In the second step, the target cues IACC, IALD, and IAPD are calculated from the variance terms as follows: TIFF2024542311000042.tif34167TIFF2024542311000043.tif69159Also, the left and right ear gains are as follows. TIFF2024542311000044.tif38165From these target cues, the final efficient synthesis of the binaural signal can be performed by designing four filters that transform the input sound into a rendered binaural output, as described in WO 2021 / 180935.
[0073] The first aspect concerns the use of basic spatial sectors. This first aspect concerns the storage of the data of the basic spatial sectors in a look-up table, the basic spatial sectors being distributed over a sphere. The data of the basic spatial sectors is preferably linked to the user's head forming a user-centric audio scene, and is the same for each tilt of the head in the same position and for each position of the listener's head, i.e., for each degree of freedom of 6-DOF. However, each movement or tilt of the head leads to a situation where the sound from the SESS "enters" the user's head in another basic spatial sector or sectors. The renderer determines the basic spatial sectors covered by the SESS, retrieves the stored data for these specific sectors, performs weighting of the stored data due to occluding objects or specific distances if necessary, then combines the stored data (or the weighted stored data, in case of weighting), and then uses the result of the combining operation for rendering (e.g., the rendering cues are calculated from the combined (co)variance data, but other steps and parameters can also be used here. Thus, this aspect may or may not use reference to occluding objects, and may or may not use reference to specific stored variance data, since the combining (and optionally also the weighting) can also be done if other data is stored, such as the (average) HRTFs (for the basic spatial sectors or for the entire spatial range) or the frequency-dependent cues themselves.
[0074] The second aspect relates to modifying objects, which may be occluding objects or other objects, so that the sound of the SESS is modified on the way from the SESS position to the user with a certain position and / or inclination. This second aspect relates to the processing of occluding objects, for example. The effect of the occluding object is a frequency-dependent attenuation with low-pass characteristics. The frequency-dependent weighting can also be applied to the prior art procedures when they do not have basic spatial sectors. Based on the transmitted data describing the occluding object, the SESS must decide whether it is occluded or not, and then apply the occlusion function to the stored cues, for example frequency-dependent, already given for different frequencies in the prior art. This is therefore a useful application of the occlusion effect in the prior art, without using basic spatial sectors or without using stored distribution data.
[0075] A third aspect relates to the storage of variance and covariance data of HRTFs, for example, for different spatial ranges or basic spatial sectors. This third aspect relates to the storage of variance and covariance data of HRTFs, for example, in a storage location, for example, in a look-up table. It is not important whether this data is stored for a certain spatial range, as in the prior art, or for basic spatial sectors. The renderer then calculates all rendering cues on the fly from the stored variance data. In contrast to the prior art applications, where at least the IACC is stored and possibly other cues or HRFT data, this is not done in this aspect. The covariance data is stored and the cues are calculated on the fly. This aspect may or may not use basic spatial sectors, and may or may not use corrections or occluding objects.
[0076] All aspects can be used separately from each other, together with each other, or only any two selected aspects can be combined.
[0077] The advantages of the present invention include, for example: Organizing the lookup table for the target cue calculation in a specific way (sector-based, use of (co)variance terms, frequency-dependent); or (Frequency selective) weighting of the (co)variance terms according to the desired target frequency response, as required for synthesis of (partially or completely) shielded parts of the SESS, or to model constant distance attenuation. The aim of the present invention is to provide an enhanced, efficient and realistic binaural rendering for spatially extended sound sources compared to WO 2021 / 180935.
[0078] Embodiments of the present invention extend the aforementioned concepts from WO 2021 / 180935 to efficiently render SESS in several ways, improving storage efficiency and enabling the ability to render even partially occluded portions of the SESS.
[0079] A particularly efficient way of organizing the look-up table and the target cue calculation based on the look-up table is disclosed, which allows covering all possible spatial target areas of the SESS with a small size of the look-up table. This is achieved by organizing the look-up table as a table dividing the whole sphere around the listener's head into small azimuth / elevation sectors. The size of these sectors (i.e. their azimuth and elevation size) is preferably selected according to the resolution of human azimuth / elevation perception. For example, the human hearing resolution for azimuth is finest in the front (about 1 degree) and decreases towards the sides. Also, the resolution of elevation perception is much coarser than the azimuth resolution, since the listener's ears are located on the left and right of the head. For each of these spatial sectors, certain partially summed terms are stored in the look-up table. In a preferred embodiment, these are the (co)variance terms (E{Yl·Yr *}, E{|Yl|2}, E{|Yr| 2 Moreover, in a preferred embodiment, these table entries are stored in a frequency selective manner (E{Yl·Yr *}, E{|Yl| 2}, E{|Yr| 2}).
[0080] This is done by subtracting these sum terms (E{Yl Yr *}, E{|Yl| 2}, E{|Yr| 2}), either alone or in addition to the above.
[0081] Furthermore, spatial weighting of specific spatial sectors (e.g., to model the shielding of this part of the SESS) can be achieved by weighting the (co)variance data stored for these spatial sectors before using them in the subsequent queue calculation process. Specifically, all (co)variance terms are weighted by the corresponding energy scaling factor g 2A desired target frequency response g(f) can be imposed by multiplying with (f). As an example, an occluding bush imposes an attenuation and low-pass frequency response when sound propagates through this occluding bush. Thus, the (co)variance terms are attenuated and higher frequency terms are attenuated more than lower frequency terms. Several zones for different occlusion / weighting are possible. Similarly, modeling of object distance is also possible: for a large object such as a river, parts of the object may be substantially farther away from the listener than other parts and therefore less loud than nearby parts. This can be modeled and rendered by distance weighting of different spatial sectors. Terms in a spatial sector are weighted with a distance energy attenuation coefficient corresponding to the (e.g., average) distance of the object in this spatial sector.
[0082] A summary of embodiments of the method or apparatus or computer program of the present invention is provided below.
[0083] During the renderer initialization / startup phase, a division of the sphere around the listener's head is performed by defining spatial sectors (e.g. azimuth and elevation ranges) into which the HRIR contributions can later be summed. Based on these spatial sectors, the corresponding HRIR contributions can then be stored in a look-up table using (co)variance terms.
[0084] Fig. 11 shows a further overview of the invention (method or device or computer program) implementing the cooperation of the first and second aspects. In particular, the block "Selection of spatial sectors for SESS rendering" corresponds to the sector identification processor 4000 shown in Figs. 1 to 3. The result of the selection of spatial sectors is a group of spatial sectors, some sectors may exist without any modification shown in 4010. Furthermore, among the determined sectors, sectors with occlusion modification according to the first characteristic shown in 4020 may be included. Furthermore, sectors with another occlusion modification shown as "number N" may also exist. This is shown in 4030. The specific target data calculation shown by the target data calculator 5000, especially for the second aspect, performs a sum of the left variance term, the right variance term and the covariance term of all non-occluded sectors, when there are two or more such sectors. Additionally, summation with weight function 1 is performed, i.e., if there are two or more sectors with occlusion / modification number 1, they are summed and then the corresponding weights are applied, or the weight and sum operations can be swapped. Additionally, if there are other sectors with occlusion modification number N as shown at 4030, such sectors can be summed with the corresponding weights of the particular weight / modification function of these sectors.
[0085] Of course, it may be the case that there are only unobstructed sectors with respect to SESS, or only obscured sectors with a single correction function, or any mixture between these possibilities, i.e., one sector is unobstructed, there is only one sector with obscured / correction number 1, but there are no sectors with obscured / correction number N. Of course, the number "N" may be equal to 1, whereby only lines 4010 and 4020 are present, but corrections having another correction in addition to correction number 1 are not determined by block 4000.
[0086] Once the individual weighting of the individual occlusions / modifications has been performed in block 5020, an overall cue summation in block 5040 is performed, followed by the input data for the final target cue calculation 5060. This target cue data is then input to the binaural cue synthesis or audio processor block 3000 of FIG. 11. The inputs to block 3000 are SESS input signal number 1 and SESS input signal number 2, if the SESS has a stereo waveform signal. In case of a SESS with only a mono waveform signal, two signals are nevertheless generated, but using a decorrelation device as shown in 3100 of FIG. 13 or 3010 of FIG. 10.
[0087] FIG. 12 shows a preferred embodiment of the binaural cue synthesis 3000 consisting of an IACC adjustment 3200, an IAPD adjustment 3300, and an IALD adjustment 3400. All these blocks are provided with data from a storage shown as "Look-up Table" in block 2000. However, depending on the implementation, the corresponding processes for determining the final values of IACC, IAPD, and IALD are also generated in block 2000 according to the target data calculation steps 5020, 5040, 5060. Therefore, the block entitled "Look-up Table" in FIG. 12 is given the reference numerals 2000 and 5000. However, the input to this block is provided by the sector identification processor 4000 of any of FIGS. 1, 2a, 3, 11.
[0088] On the left side of Fig. 13, a decorrelator 3100 is shown for generating from a single SESS waveform signal two SESS input signals numbered 1 and numbered 2 at the output of the decorrelator. This data is then subjected to four filtering operations 3210, 3220, 3230 and 3240, with the corresponding contributions of the left channel being added via adder 3250 and the corresponding contributions of the right channel being added via adder 3260 to obtain the final left and right output signals. The individual filter functions 3210, 3220, 3230 and 3240 are calculated via a target data calculator 5000 for a correspondingly determined limited spatial range, as described in WO 2021 / 180935, or according to a number of elementary spatial sectors, as explained in relation to Fig. 7, where the spatially extended sound source is represented by more than one elementary spatial sector.
[0089] The processing of each audio block is illustrated in Fig. 11, which shows an overall flow chart of a preferred embodiment implementing the first, second and third aspects together. For each audio signal block, the (time-varying) target cues of the target spatial domain belonging to SESS are determined and applied to the two input signals in a binaural cue synthesis stage to generate the L and R binaural output signals.
[0090] The target binaural cues are calculated as follows: Taking into account the positions and orientations of the listener and the SESS, as well as the SESS geometry, the spatial sector belonging to the SESS is calculated (eg, using a projection algorithm or ray tracing analysis).
[0091] Specifically, spatial sectors are found that belong to the parts of the SESS that should be weighted to model effects such as occlusion and / or distance attenuation, etc. There may be several spatial regions that require different attenuation / frequency response characteristics. The corresponding sectors are processed separately with each region belonging to a different so-called "sector class" (e.g. "unoccluded", "occluded / modified #1", ... "occluded / modified #n").
[0092] The (co)variance terms stored for the sectors within each sector class are summed. The total sector (co)variance data of the different sector classes are then weighted according to the desired transfer function for each sector class. Specifically, the (co)variance data of that sector class is multiplied by the (frequency-dependent) energy transfer function (amplitude scaling factor / amplitude frequency response squared) belonging to this class.
[0093] The weighted variance terms of all sector classes of the SESS are summed into an overall (weighted) (co)variance term.
[0094] The target cue using the modified / weighted overall (co)variance terms is calculated using equations (23)-(27). Of course, the (co)variance data for each sector can also be weighted and then summed individually, rather than first performing partial sums within the sector class, weighting once for each sector class, and finally summing. However, the aforementioned approach is the preferred embodiment due to its higher efficiency.
[0095] Advantages of embodiments of the present invention over the state of the art include very efficient and more realistic rendering of Sized Sound Sources (SESS), small lookup table sizes, and / or the ability to include rendering effects (such as partial occlusion or distance attenuation) that change the frequency response of selected spatial portions of the SESS.
[0096] A preferred example relates to a renderer that uses as input one or more signal channels, the geometry, size and orientation of a spatially extended sound source (SESS) and a set of HRTFs, and is equipped for binaural rendering of the spatially extended sound source (i.e. providing two output signals).
[0097] Further preferred renderers or apparatus and methods for synthesizing SPESS additionally or instead of the above comprise a target cue calculation stage (e.g. for calculating the desired interaural target cues) and a cue synthesis stage (e.g. for converting the input signal(s) into a binaurally rendered signal having the desired target cues).
[0098] Further preferred renderers or devices and methods for synthesizing SPESS additionally or alternatively include pre-calculated data for the binaural rendering of SESS and include the use of look-up tables provided / pre-calculated for different frequency bands depending on the HRTF set.
[0099] Further preferred renderers or apparatus and methods for synthesizing SPESS additionally or instead of the above comprise a look-up table organized to store (co)variance terms for each spatial sector (e.g., l (left) variance, r (right) variance, lr covariance).
[0100] In another preferred embodiment, the spatial sector is defined as an azimuth / elevation range.
[0101] In other preferred embodiments, the spatial sector size is selected with respect to the resolution of human auditory spatial localization capabilities (eg, wider in elevation than in azimuth).
[0102] In another preferred embodiment, the calculation of the target binaural rendering cues is performed based on the summed variance terms of the spatial sectors belonging to the SESS.
[0103] In another preferred embodiment, modification of the rendering of different spatial regions of the SESS (e.g., occlusion or distance modeling) is achieved by using modified variance terms from a lookup table rather than the originally stored ones.
[0104] In another preferred embodiment, the modification is performed by multiplication of the dispersion term with an energy attenuation coefficient belonging to the spatial sector.
[0105] In other preferred embodiments, the attenuation coefficient is frequency dependent (eg, to model the low-pass effect due to partial occlusion).
[0106] A further embodiment relates to a bitstream that contains the following information: size, position and orientation of objects and waveforms, and geometry of occluding objects.
[0107] This is followed by a description of a further preferred embodiment currently being developed for MPEG I ISO 23090-4.
[0108] This embodiment synthesizes one or more spatially extended sound sources (SESS) for reproduction in headphones for object sources that have an associated flag objectSourceHasExtent set to 1. The parameters of each of the object sources are identified by objectSourceExtentId.
[0109] The synthesis is based on the SESS description with (ideally) infinite uncorrelated point sources distributed over the entire spatial extent of the source range. By continuously projecting the SESS geometry in a direction towards the current listener position, the range covered by said geometry can be identified for each frame and updated in real time. In other words, the geometry is projected onto a sphere representing the user's virtual listening space for each frame. Then, the spatial section occupied by the projected geometry on the sphere is what is included in the auralization of SESS.
[0110] The SESS is defined by the user in the Encoder Input Format (EIF). Given a desired source range, the SESS is synthesized using two uncorrelated input signals. These input signals are processed in such a way that perceptually significant auditory cues are synthesized. This includes the following interaural cues: Interaural Cross-Correlation (IACC), Interaural Phase Difference (IAPD) and Interaural Level Difference (IALD). In addition, monophonic spectral cues are reproduced. This is shown in Figure 12.
[0111] Data Elements and Variables itemStore A local pointer to a RenderItemStore object. B Block size Fs Sampling rate extentProcessors A mapping from item id to its extentProcessor instance. extentDownmixItem RI that stores the final output of the binaural signal for all ranges. Stage Description To save real-time computational costs, individual HRTF points are assigned to a predefined grid table that separates the listener's virtual listening sphere into uniformly distributed regions. During initialization, an N-point DFT is performed to obtain N / 2+1 frequency components for each HRIR, where N is its length. Then, we integrate the data of all HRTF points in the unnormalized IACC, which are the gains of the left and right channels, to obtain three intermediate values for each grid. In addition, we also store the number of HRTF data points contained in each grid. These are used to calculate the final cues in real time.
[0112] The gains for both channels of each grid are calculated by Equations 28 and 29, where: TIFF2024542311000045.tif78 and TIFF2024542311000046.tif79 are the magnitudes of the left and right HRTFs respectively, and N is the number of HRTF points within this grid. TIFF2024542311000047.tif735(28) TIFF2024542311000048.tif736(29) The unnormalized IACC for each grid is calculated using Equation 30, where,φ,l,and,φ,r,are the phases of the left and right HRTFs, respectively. TIFF2024542311000049.tif780(30) The procedures of equations 28 to 30 are performed in advance before the actual processing and correspond to steps 800, 810 in FIG. 8, and the results of these processing are data that are preferably stored in storage device 2000 or 200 in the corresponding figures.
[0113] During real-time processing, each unique augmentation source is generated and managed by an augmentation processor. Each active processor receives, for each frame, a buffer of audio samples and metadata that indicates how to synthesize the augmentation source. There are two separate processing chains: metadata processing in the update thread and audio processing in the audio thread. These are described in the following sections respectively, and their results are combined at the end of the second chain to generate the binaural audio output.
[0114] Calculations performed in the update thread: For each unique extended sound source, one or more metadata carriers in the form of RI (Rendering Item) are generated by an occlusion stage (corresponding for example to block 4000).
[0115] This stage 4000 loops through all incoming RIs and assigns the relevant range metadata to the corresponding processor. If one of the spatial sections from the predefined table is covered and should be included to auralize the range in this frame, the incoming metadata includes a gain factor (items 4010, 4020, 4030 in FIG. 11) and a list of gains corresponding to some predefined frequency bins for it. By selecting (e.g. 4000), weighting (e.g. 5020) and finally accumulating (e.g. 5040) the intermediate data stored with gain and EQ, the generation of an extended sound source of any shape with any form and degree of obscuration (size / material) is achieved.
[0116] The final filter is obtained by the following steps: After integrating (or accumulating) all grid points shown in the RI (rendering item), the gains and IACC (i.e. variance and covariance data) of the left and right channels are normalized by the weighted total number of HRTF data points. TIFF2024542311000050.tif1339(31) TIFF2024542311000051.tif1340(32) TIFF2024542311000052.tif1353(33) The procedure of Equation 31 to Equation 33 corresponds to block 5040 .
[0117] Frequency Dependent TIFF2024542311000053.tif78 and TIFF2024542311000054.tif78 is calculated using the normalized IACC. TIFF2024542311000055.tif1376(34) TIFF2024542311000056.tif1348(35) The calculations at block 5060 correspond to the processing of Equations 34 and 35 in one embodiment.
[0118] The final stereo filters 3210, 3220, 3230, and 3240 are TIFF2024542311000057.tif78 and TIFF2024542311000058.tif78, left and right channel gain ( TIFF2024542311000059.tif75 and TIFF2024542311000060.tif75), and the phase extracted from the HRTF points corresponds to the center of the range. ( TIFF2024542311000061.tif714 and TIFF2024542311000062.tif715) TIFF2024542311000063.tif773(36) TIFF2024542311000064.tif780(37) TIFF2024542311000065.tif779(38) TIFF2024542311000066.tif781(39) The calculations of blocks 36 through 39 are preferably also performed in block 5060.
[0119] Calculations performed in the audio thread: The input mono signal is first fed into a decorrelator 3100 to obtain two decorrelated versions. An MPEG-I decorrelator or any other decorrelator such as that shown in FIG. 10 can be used.
[0120] Each of the two decorrelated signals is then convolved with a corresponding stereo filter 3210, 3220, 3230, 3240 calculated in the update thread, resulting in four output channels. Cross-mixing 3250, 3260 is then performed to generate the final binaural output.
[0121] Equations (40) and (41) define the (filtering and) mixing process, where: TIFF2024542311000067.tif75 and TIFF2024542311000068.tif75 represents two uncorrelated signals, TIFF2024542311000069.tif76 and TIFF2024542311000070.tif76 are the two stereo filters (one for left and one for right) calculated in the metadata processing section. Figure 13 is a signal flow diagram of the process. The filters shown in Figure 13 are similar to those in Figure 9. TIFF2024542311000071.tif790(40) TIFF2024542311000072.tif792(41) The processing according to Equation 40 and Equation 41 is preferably implemented in an audio processor or binaural cue synthesis block, 3000 in FIG. 11 or 300 in FIGS.
[0122] FIG. 7 shows a schematic diagram of the rendering range of a listener. The rendering range is illustratively a sphere centered on the user. The user or listener (not shown in FIG. 7) is therefore located at the center of the sphere, and the rendering range corresponding to this sphere around the listener can be considered as being "tied" to the user's hand. Thus, when the user changes his / her position either horizontally, vertically or in depth (x, y, z), the sphere moves around according to the user's movement relative to a spatially extended sound source that can be considered fixed relative to the user. Furthermore, when the user moves his / her hand by looking up, down or to the side, the sphere representing the listener's rendering range also moves up, down or to the side, i.e., it also performs the "movement" that the user applies to his / her head without moving horizontally, vertically or in depth. The listener's spherical rendering range can therefore be considered as a kind of "helmet" that always follows the movement of the user's / listener's head in all six degrees of freedom.
[0123] The sphere is separated into individual basic spatial sectors that can be spaced apart and therefore have different dimensions in azimuth and elevation to reflect psychoacoustic findings. In particular, the rendering range includes a sphere or a portion of a sphere around the listener, and each basic spatial sector shown in FIG. 7 has, for example, an azimuth size and an elevation size. In particular, the azimuth size and the elevation size of the basic spatial sectors are different from each other, so that the azimuth size is finer for a basic spatial sector in front of the listener compared to the azimuth size of a basic spatial sector that is more to the side of the listener, and / or the azimuth size decreases towards the side of the listener, and / or the elevation size of a basic spatial sector is smaller than the azimuth size of this sector.
[0124] Aspects of the invention therefore rely on a user-centric representation of a spatially extended sound source that moves with the user, with the user's head at the centre of the space and a sphere or part of a sphere being the rendering range.
[0125] Now, the sector identification processor 4000 determines which different basic spatial sectors represent the spatially extended sound source shown in FIG. 7 as 7000. In this example, for example, by a ray tracing algorithm starting from the center of this sphere and pointing to the SESS 7000, four basic spatial sectors ESS, shown in FIG. 7 as "1", "2", "3" and "4", are determined to "belong" to the SESS 7000 at a particular orientation and position of the user relative to the SESS 7000. It is therefore assumed that the sound field emitted by the SESS 7000 that actually reaches the user's ear passes through these four ESSs. Furthermore, an occluding object 7010 is also shown in FIG. 7, and for the purposes of the example, it is assumed that the basic spatial sector (ESS1) is completely occluded, the basic spatial sector 2 (ESS2) is partially occluded, and ESSs 3, 4 are not occluded by the occluding object.
[0126] Thus, referring to Figure 11, base space sectors 1 and 2 correspond to item 4010, base space sector 1 corresponds to item 4020, and base space sector 2 corresponds to item 4030 in Figure 11. Alternatively, partially occluded sectors can be determined to belong to the same class as fully occluded sectors, or sectors with occlusion below a certain threshold can also be determined to be completely unoccluded if the sector is only very small in part occluded.
[0127] 7 shows that the basic spatial sectors and the optional degree of occlusion of the sector occlusion or modification characteristics are the same for both ears, i.e., left and right, but the number and / or identity of the basic spatial sectors may be different for the left and right ears. This may easily be the case when the SESS is very close to the user and the SESS is located centrally between both ears rather than one or the other.
[0128] Furthermore, procedures other than ray tracing algorithms can be performed to determine the projection of the SESS onto the rendering range for the listener, i.e., for the exemplary sphere. Note that the SESS 7000 does not necessarily have to be fixed. The SESS can also be dynamic, i.e., move over time. Then, the SESS position relative to the user needs to be determined in advance, and then for a certain time / frame of the SESS waveform signal, the corresponding basic spatial sectors on the left and right sides of the listener relative to the actual position of the listener's head are determined, and then the cues are calculated as shown for logs 5020 to 5060 in FIG. 11.
[0129] Furthermore, it should be noted here that the rendering range does not necessarily have to be a full sphere. The rendering range may include only a portion of a sphere. Furthermore, the rendering range does not necessarily have to be spherical. The rendering range may be cylindrical or have a polygonal shape, as long as it covers a certain three-dimensional portion of the space around the listener.
[0130] With regard to the size of the basic spatial sector, it should be emphasized that the basic spatial sector can be very small and for the determination of the stored rendering data items only a single HRTF represented in amplitude and phase instead of summing over a certain number (e.g. as shown in Equations 20, 21 and 22 or Equations 28 to 30 is sufficient). However, when a basic spatial sector with a certain dimension is used, so that the size of the storage device storing the rendering data items of each basic spatial sector is reduced, the determination of the rendering data items stored in the storage device of each basic spatial sector can be performed according to Equations 20 to 22 or Equations 28 to 30, and the HRTFs belonging only to a certain basic spatial sector are summed to obtain the specific frequency and actual (co)variance data of this basic spatial sector.
[0131] It should be noted that a particular advantage of this procedure is that it is not necessary to perform all of these calculations at runtime: instead, the stored data for each or each basic space sector can be calculated and stored as soon as a particular division of the rendering range into a particular grid or grid points of the basic space sector is determined, and for a particular initialization on a particular grid, the only procedure to be performed during runtime is to load the corresponding pre-calculated data of this grid into a storage device or look-up table.
[0132] The only procedure that needs to be performed during runtime is the identification of the basic spatial sectors belonging to the spatially extended sound sources for a particular user orientation / position and potentially necessary weighting due to occluding objects, then the final overall summation corresponding to block 5040 of Fig. 11 gives a free method for the final target cue calculation in block 5060. Thus, the computation operations required during runtime are very limited and very small compared to the computation operations required to determine the basic spatial sectors, i.e. the rendering data items of a particular grid.
[0133] Furthermore, it should be noted that the storage of a particular grid is independent of the user's position / orientation, since if the location or characteristics of the SESS change, or if the user's orientation / position changes, only the identified basic spatial sector changes, but the data stored for the basic spatial sector representing the grid does not change. In other words, only the ID number of the basic spatial sector changes, and the data for the basic spatial sector with a particular ID number does not change.
[0134] Next, FIG. 8 will be described to illustrate a preferred procedure according to one or more embodiments of the present invention.
[0135] In step 800, a rendering range, such as a sphere, is determined or initialized. The result is, for example, a sphere with particular grid points or base space sectors. In block 810, rendering data items, such as (co)variance data, are stored in a storage device, such as a look-up table, for all base space sectors within the rendering range.
[0136] Next, in step 820, sector identification is performed as done by block 4000. Thus, one or more basic spatial sectors belonging to a spatially extended sound source are determined based on the listener's SESS data and position / orientation data input to block 820. The result of block 820 is one or more basic spatial sectors.
[0137] At block 830, a summation of the rendering data items for multiple base spatial sectors, with or without weighting, is performed as indicated by block 5040.
[0138] In block 840, target rendering data such as IACC, IALD, IAPD, GL, and GR are calculated, which is performed by block 5060.
[0139] In block 850, the target rendering data is applied to the spatially extended source audio signal as shown, for example by the audio processor block 3000 or also the binaural cue synthesis block 3000 of FIG.
[0140] According to the first aspect of the present invention, the rendering sphere is implemented as shown in Fig. 7, i.e. the basic spatial sectors covering the rendering range of the listener are determined, and the sector identification processor defines a set of basic spatial sectors, such as two or more basic spatial sectors of the spatially extended sound source. However, it is only a preferred embodiment that the stored rendering data items are variance or covariance data. Alternatively, other data items required for rendering can also be stored and combined by the target data calculator. This procedure also does not necessarily require correction processing, although it is preferable to do so.
[0141] According to a second aspect of the present invention, a determination of potential modified objects and a determination of limited modified space sectors based on potential modified object identification are required. However, for this procedure, the rendering range does not necessarily have to be dimensioned as shown in FIG. 7, i.e., each elementary space sector has an individual stored data item. Instead, the rendering range can also be implemented as shown in other embodiments, such as those shown in WO 2021 / 180935. Furthermore, for the determination and consideration of modified objects, the stored rendering data items are not necessarily variance / covariance data. Instead, other rendering data, such as those shown as stored data in WO 2021 / 180935, can be used as well.
[0142] With regard to the third aspect, the determination of the rendering range as shown in Fig. 7 is not necessarily required. Instead, other determinations can be used for one or more restricted spatial sectors, such as the definition of the rendering range as shown in WO 2021 / 180935. However, the restricted spatial sectors are preferably implemented as the basic spatial sectors shown in Fig. 7. Furthermore, specific processing of modified / occluded objects, for the purpose of using variance / covariance data as stored data, is also not a mandatory feature, but is preferred, for example as described above with respect to block 830 in Fig. 8.
[0143] Further embodiments related to the first aspect are summarized subsequently. An embodiment relates to an apparatus for synthesizing a spatially extended sound source (SESS), comprising: a storage device for storing rendering data items for different basic spatial sectors covering a rendering range for a listener; a sector identification processor for identifying a set of basic spatial sectors belonging to the spatially extended sound source from the different basic spatial sectors based on listener data and data of the spatially extended sound source; a target data calculator for calculating target rendering data from the rendering data items for the set of basic spatial sectors; and an audio processor for processing an audio signal representing the spatially extended sound source using the target rendering data.
[0144] In a further embodiment, the storage device is configured to store, for each basic spatial sector as a rendering data item, at least one of a left variance data item relating to left head-related transfer function data, a right variance data item relating to right head-related transfer function (HRTF) data, and a covariance data item relating to left HRTF data and right HRTF data, the target calculator is configured to sum the left variance data items of the set of basic spatial sectors or the right variance data items of the set of basic spatial sectors or the covariance data items of the set of basic spatial sectors, respectively, to obtain at least one summed item, the target calculator is configured to calculate at least one rendering cue as target rendering data from the at least one summed item, and the audio processor is configured to process the audio signal using the at least one rendering cue.
[0145] In further embodiments, the sector identification processor is configured to apply a projection algorithm or ray tracing analysis to determine the set of basic spatial sectors, or uses listener position or listener orientation as listener data, or uses information about listener orientation, listener position, or listener geometry as spatially extended sound source (SESS) data.
[0146] In a further embodiment, the sector identification processor is configured to receive occlusion information for potentially occluded objects from a description of the audio scene and to determine, based on the occlusion information, a particular spatial sector of the set of basic spatial sectors as an occluded sector, and the target data calculator is configured to apply an occlusion function to the rendering data items stored for the occluded sector to obtain modified data and to use the modified data to calculate the target rendering data.
[0147] In a further embodiment, the occlusion function is a low pass function having different attenuation values for different frequencies and the rendering data items are data items of different frequencies and the target data calculator is configured for some frequencies to weight the data items of particular frequencies with the attenuation values of the particular frequencies to obtain modified rendering data.
[0148] In a further embodiment, the sector identification processor is configured to determine that another basic spatial sector of the set of basic spatial sectors determined for the occluding object is not occluded by the potentially occluding object, and the target data calculator is configured to combine the modified data from the occluded sector with the rendering data items of the other sectors, either without modification with the occlusion function or modified by a different modification function, to obtain the target rendering data.
[0149] In a further embodiment, the sector identification processor is configured to determine a first basic spatial sector of the set of basic spatial sectors as having a first characteristic and to determine a second basic spatial sector of the set of basic spatial sectors as having a second, different characteristic, and the target data calculator is configured to apply no modification function to the first basic spatial sector and to apply a modification function to the second basic spatial sector, or to apply the first modification function to the first basic spatial sector and to apply a second modification function to the second basic spatial sector, the second modification function being different from the first modification function.
[0150] In further embodiments, the first correction function is frequency selective and the second correction function is constant over frequency, or the first correction function has a first frequency selective characteristic and the second correction function has a second frequency selective characteristic different from the first frequency selective characteristic, or the first correction function has a first attenuation characteristic and the second correction function has a second different attenuation characteristic, and the target data calculator is configured to select or adjust the correction function from the first and second correction functions based on a distance between the first or second basic spatial sector and the listener, or based on characteristics of an object located between the listener and the corresponding basic spatial sector.
[0151] In a further embodiment, the sector identification processor is configured to classify the set of basic spatial sectors into different sector classes based on characteristics associated with the basic spatial sectors, and the target data calculator is configured to combine rendering data items of the basic spatial sectors of each class to obtain a combined result of each class if two or more basic spatial sectors are in a class, and to apply a specific correction function associated with at least one class to the combined result of this class to obtain a modified combined result of this class, or to apply a specific correction function associated with at least one class to one or more data items of one or more basic spatial sectors of each class to obtain a modified data item, combine the modified data items of the basic spatial sectors of each class to obtain a modified combined result of this class, combine the combined result or, if available, the modified combined result of each class to obtain an overall combined result, and use the overall combined result as the target rendering data, or to calculate the target rendering data from the overall combined result.
[0152] In a further embodiment, the characteristics of the basic spatial sector are determined to be one of a group including an occluded basic spatial sector including a first occlusion characteristic, an occluded basic spatial sector including a second occlusion characteristic different from the first occlusion characteristic, an unoccluded basic spatial sector having a first distance to the listener, and an unoccluded basic spatial sector having a second distance to the listener, the second distance being different from the first distance.
[0153] In a further embodiment, the target data calculator is configured to modify or combine the frequency dependent variance or covariance parameters as the rendering data items to obtain an overall combined variance or overall combined covariance parameter as an overall combination result, and calculate at least one of the interaural coherence cue, the interaural level difference cue, the interaural phase difference cue, the first side gain, or the second side gain as the target rendering data.
[0154] In a further embodiment, the audio processor is configured to perform at least one of inter-channel coherence adjustment, inter-channel phase difference adjustment, and inter-channel level difference adjustment using the corresponding cue as target rendering data.
[0155] In a further embodiment, the rendering range comprises a sphere or a portion of a sphere around the listener, the rendering range being associated with the listener's position or listener's orientation, and each elementary spatial sector having an azimuth and an elevation size.
[0156] In further embodiments, the azimuth and elevation sizes of the basic spatial sectors differ from each other, such that the azimuth size is finer for a basic spatial sector that is in front of the listener compared to the azimuth size of a basic spatial sector that is further to the side of the listener, or the azimuth size decreases towards the side of the listener, or the elevation size of a basic spatial sector is smaller than the azimuth size of this sector.
[0157] Further embodiments relating to the second aspect are summarized subsequently. An embodiment of an apparatus for synthesizing a spatially extended sound source comprises an input interface for receiving a description of an audio scene, the audio scene description comprising spatially extended sound source data for the spatially extended sound source and modification data for potential modification objects, and for receiving listener data; a sector identification processor for identifying a restricted modified spatial sector of the spatially extended sound source within a rendering range of the listener based on the spatially extended sound source data and the listener data and the modification data, the rendering range of the listener being larger than the restricted modified spatial sector; a target data calculator for calculating target rendering data from one or more rendering data items belonging to the modified restricted spatial sector; and an audio processor for processing an audio signal representing the spatially extended sound source using the target rendering data.
[0158] In a further embodiment, the modified data is occlusion data and the potentially modified object is a potentially occluding object.
[0159] In a further embodiment, the potential modification object has an associated modification function, the one or more rendering data items are frequency dependent, the modification function is frequency selective, and the target data calculator is configured to apply the frequency selective modification function to the one or more frequency dependent rendering data items.
[0160] In a further embodiment, the frequency-selective correction function has different values for different frequencies and the frequency-dependent one or more rendering data items have different values for different frequencies, and the target data calculator is configured to apply, multiply or combine a value of the frequency-selective correction function for a particular frequency to the value of the one or more rendering data items for the particular frequency.
[0161] In a further embodiment, a storage device is provided for storing one or more rendering data items for a number of different restricted spatial sectors, which together form a rendering range for a listener.
[0162] In a further embodiment, the modification function is a frequency selective low-pass function and the target data calculator is configured to apply the low-pass function such that values of the one or more rendering data items at higher frequencies are attenuated more strongly than values of the one or more rendering data items at lower frequencies.
[0163] In a further embodiment, the sector identification processor is configured to determine a restricted spatial sector of the spatially extended sound source based on the listener data and the spatially extended sound source data, determine whether at least a portion of the restricted spatial sector is subject to modification by the modification object, and determine the restricted spatial sector as a modified spatial sector if the portion is greater than a threshold or if the entire restricted spatial sector is subject to modification by the modification object.
[0164] In further embodiments, the sector identification processor is configured to apply a projection algorithm or ray tracing analysis to determine the restricted spatial sector, or to use listener position or listener orientation as listener data, or to use information about listener orientation, listener position, or listener geometry as spatially extended sound source (SESS) data.
[0165] In a further embodiment, the rendering range comprises a sphere or a portion of a sphere around the listener, the rendering range being associated with the listener's position or listener's orientation, and the modified limited spatial sector having an azimuth and elevation size.
[0166] In further embodiments, the azimuth and elevation sizes of the modified restricted spatial sectors differ from one another, such that the azimuth size is finer for modified restricted spatial sectors in front of the listener compared to the azimuth size of modified restricted spatial sectors further to the side of the listener, or the azimuth size decreases towards the side of the listener, or the elevation size of the modified restricted spatial sector is smaller than the azimuth size of the modified restricted spatial sector.
[0167] In a further embodiment, as the one or more rendering data items, at least one of a left variance data item relating to left head-related transfer function data, a right variance data item relating to right head-related transfer function (HRTF) data, and a covariance data item relating to left HRTF data and right HRTF data is used for the modified restricted spatial sector.
[0168] In a further embodiment, the sector identification processor is configured to determine a set of basic spatial sectors belonging to the spatially extended sound source and to determine, among the set of basic spatial sectors, one or more basic spatial sectors as restricted modified spatial sectors, and the target data calculator is configured to modify one or more rendering data items associated with the restricted modified spatial sectors using the modification data to obtain combined data and to combine the combined data with rendering data items of one or more basic spatial sectors of the set of basic spatial sectors that are different from the restricted modified spatial sectors, that are unmodified or that are modified in a different manner compared to the modification of the restricted modified spatial sector.
[0169] In a further embodiment, the sector identification processor is configured to classify the set of basic spatial sectors into different sector classes based on characteristics associated with the basic spatial sectors, and the target data calculator is configured to combine rendering data items of the basic spatial sectors of each class to obtain a combined result of each class if two or more basic spatial sectors are in a class, and to apply a specific correction function associated with at least one class to the combined result of this class to obtain a modified combined result of this class, or to apply a specific correction function associated with at least one class to one or more data items of one or more basic spatial sectors of each class to obtain a modified data item, combine the modified data items of the basic spatial sectors of each class to obtain a modified combined result of this class, combine the combined result or, if available, the modified combined result of each class to obtain an overall combined result, and use the overall combined result as the target rendering data, or to calculate the target rendering data from the overall combined result.
[0170] In a further embodiment, the characteristics of the basic spatial sector are determined to be one of a group including an occluded basic spatial sector including a first occlusion characteristic, an occluded basic spatial sector including a second occlusion characteristic different from the first occlusion characteristic, an unoccluded basic spatial sector having a first distance to the listener, and an unoccluded basic spatial sector having a second distance to the listener, the second distance being different from the first distance.
[0171] In a further embodiment, the target data calculator is configured to modify or combine the frequency dependent variance or covariance parameters as the rendering data items to obtain as an overall combination result an overall combined variance or an overall combined covariance parameter, and to calculate at least one of the interaural or inter-channel coherence cue, the interaural or inter-channel level difference cue, the interaural or inter-channel phase difference cue, the first side gain, or the second side gain as the target rendering data, and the audio processor is configured to process the audio signal using at least one of the interaural or inter-channel coherence cue, the interaural or inter-channel level difference cue, the interaural or inter-channel phase difference cue, the first side gain, or the second side gain as the target rendering data.
[0172] A further embodiment comprises an audio scene generator for generating an audio scene description, the audio scene generator comprising a spatially extended sound source (SESS) data generator for generating SESS data for a spatially extended sound source, a modification data generator for generating modification data for a potential modification object, and an output interface for generating an audio scene description including the SESS data and the modification data.
[0173] In a further embodiment, the modification data comprises a description of a low pass function or geometry data for the potential modified object, the low pass function comprising attenuation values for higher frequencies representing stronger attenuation values compared to the attenuation values for lower frequencies, and the output interface is configured to introduce the description of the attenuation function or geometry data for the potential modified object as modification data into the audio scene description.
[0174] In a further embodiment, the SESS data generator is configured to generate information regarding the position of the SESS and the geometry of the SESS as the SESS data, and the output interface is configured to introduce information regarding the position of the SESS and information regarding the geometry of the SESS as the SESS data.
[0175] In further embodiments, the SESS data generator is configured to generate, as the SESS data, information regarding the size, position or orientation of the spatially extended sound source, or waveform data of one or more audio signals associated with the spatially extended sound source, or the correction data calculator is configured to calculate, as the correction data, the geometry of potential correction objects, such as potential occluding objects.
[0176] Further embodiments include an audio scene description that includes spatially extended sound source data and modification data for one or more potential modified objects.
[0177] In a further embodiment, the audio scene description is embodied as a transmitted or stored bitstream, the spatially extended sound source data representing a first bitstream element and the modification data representing a second bitstream element.
[0178] Further embodiments relating to the third aspect are summarized subsequently. An embodiment comprises an apparatus for synthesizing a spatially extended sound source (SESS), comprising: a storage device for storing one or more rendering data items for different restricted spatial sectors, the different restricted spatial sectors being located within a rendering range of a listener, the one or more rendering data items for the restricted spatial sectors comprising at least one of a left variance data item associated with left head function data, a right variance data item associated with right head function data, and a left / right covariance data item associated with the left head function data and the right head function data; a sector identification processor for identifying one or more restricted spatial sectors for a spatially extended sound source within the rendering range of the listener based on the spatially extended sound source data; a target data calculator for calculating target rendering data from the stored left variance data, the stored right variance data, or the stored covariance data; and an audio processor for processing an audio signal representing the spatially extended sound source using the target rendering data.
[0179] In a further embodiment, the storage device is configured to store head-related transfer function data, or binaural room impulse response data, or a variance data item or a covariance data item related to the binaural room transfer function data, or the head-related impulse response data.
[0180] In a further embodiment, one or more rendering data items include variance or covariance data item values of different frequencies.
[0181] In a further embodiment, the storage device is configured to store, for each restricted spatial sector, a frequency-dependent representation of a left variance data item, a frequency-dependent representation of a right variance data item, and a frequency-dependent representation of a covariance data item.
[0182] In a further embodiment, the target data calculator is configured to calculate at least one of an interaural or inter-channel coherence cue, an interaural or inter-channel level difference cue, an interaural or inter-channel phase difference cue, a first side gain, and a second side gain as the target rendering data, and the audio processor is configured to perform at least one of an inter-channel or interaural coherence adjustment, an interaural or inter-channel phase difference adjustment, or an interaural or inter-channel level difference adjustment using the corresponding cue as the target rendering data.
[0183] In further embodiments, the target data calculator is configured to calculate interaural or inter-channel coherence cues based on the left variance data item, the right variance data item, and the covariance data item, or to calculate inter-channel or interaural phase difference cues based on the left variance data item and the right variance data item, or to calculate inter-channel or interaural phase difference cues based on the covariance data item, or to calculate left or right side gains using the left or right variance data item and information regarding the signal power of the audio signal.
[0184] In further embodiments, the target data calculator is configured to calculate the interaural or inter-channel coherence cues such that their values are within a range of + / -20% of the value obtained by the interaural or inter-channel coherence cues equations described herein; or the target data calculator is configured to calculate the interaural or inter-channel level difference cues such that their values are within a range of + / -20% of the value obtained by the interaural or inter-channel level difference cues equations described herein; or the target data calculator is configured to calculate the interaural or inter-channel phase difference cues such that their values are within a range of + / -20% of the value obtained by the interaural or inter-channel phase difference cues equations described herein; or the target data calculator is configured to calculate the first or second side gains such that their values are within a range of + / -20% of the value obtained by the left or right side gain equations described herein.
[0185] In further embodiments, the sector identification processor is configured to apply a projection algorithm or ray tracing analysis to determine one or more restricted spatial sectors as a set of basic spatial sectors, or to use listener position or listener orientation as listener data, or to use information on the listener orientation, the listener position, or the listener geometry as spatially extended sound source (SESS) data.
[0186] In further embodiments, the rendering range comprises a sphere or portion of a sphere around the listener, the rendering range being associated with the listener's position or listener's orientation, and the one or more bounded spatial sectors having an azimuth and elevation size.
[0187] In further embodiments, the azimuth and elevation sizes of the different restricted spatial sectors differ from one another, such that the azimuth size is finer for a different restricted spatial sector that is in front of the listener compared to the azimuth size of a different restricted spatial sector that is further to the side of the listener, or the azimuth size decreases towards the side of the listener, or the elevation size of a restricted spatial sector is smaller than the azimuth size of that sector.
[0188] In a further embodiment, the sector identification processor is configured to determine a set of basic spatial sectors as one or more restricted spatial sectors, and for each basic spatial sector at least one of a left variance data item, a right variance data item, and a covariance data item is stored.
[0189] In a further embodiment, the sector identification processor is configured to receive occlusion information for potentially occluded objects from a description of the audio scene and to determine, based on the occlusion information, a particular spatial sector of the set of basic spatial sectors as an occluded sector, and the target data calculator is configured to apply an occlusion function to the rendering data items stored for the occluded sector to obtain modified data and to use the modified data to calculate the target rendering data.
[0190] In a further embodiment, the occlusion function is a low pass function having different attenuation values for different frequencies and the rendering data items are data items of different frequencies and the target data calculator is configured for some frequencies to weight the data items of particular frequencies with the attenuation values of the particular frequencies to obtain modified rendering data.
[0191] In a further embodiment, the sector identification processor is configured to determine that another basic spatial sector of the set of basic spatial sectors determined for the occluding object is not occluded by the potentially occluding object, and the target data calculator is configured to combine the modified data from the occluded sector with the rendering data items of the other sectors, either without modification with the occlusion function or modified by a different modification function, to obtain the target rendering data.
[0192] In a further embodiment, the sector identification processor is configured to determine a first basic spatial sector of the set of basic spatial sectors as having a first characteristic and to determine a second basic spatial sector of the set of basic spatial sectors as having a second, different characteristic, and the target data calculator is configured to apply no modification function to the first basic spatial sector and to apply a modification function to the second basic spatial sector, or to apply the first modification function to the first basic spatial sector and to apply a second modification function to the second basic spatial sector, the second modification function being different from the first modification function.
[0193] In further embodiments, the first correction function is frequency selective and the second correction function is constant over frequency, or the first correction function has a first frequency selective characteristic and the second correction function has a second frequency selective characteristic different from the first frequency selective characteristic, or the first correction function has a first attenuation characteristic and the second correction function has a second different attenuation characteristic, and the target data calculator is configured to select or adjust the correction function from the first and second correction functions based on a distance between the first or second basic spatial sector and the listener, or based on characteristics of an object located between the listener and the corresponding basic spatial sector.
[0194] In a further embodiment, the sector identification processor is configured to classify the set of basic spatial sectors into different sector classes based on characteristics associated with the basic spatial sectors, and the target data calculator is configured to combine rendering data items of the basic spatial sectors of each class to obtain a combined result of each class if two or more basic spatial sectors are in a class, and to apply a specific correction function associated with at least one class to the combined result of this class to obtain a modified combined result of this class, or to apply a specific correction function associated with at least one class to one or more data items of one or more basic spatial sectors of each class to obtain a modified data item, combine the modified data items of the basic spatial sectors of each class to obtain a modified combined result of this class, combine the combined result or, if available, the modified combined result of each class to obtain an overall combined result, and use the overall combined result as the target rendering data, or to calculate the target rendering data from the overall combined result.
[0195] In a further embodiment, the characteristics of the basic spatial sector are determined to be one of a group including an occluded basic spatial sector including a first occlusion characteristic, an occluded basic spatial sector including a second occlusion characteristic different from the first occlusion characteristic, an unoccluded basic spatial sector having a first distance to the listener, and an unoccluded basic spatial sector having a second distance to the listener, the second distance being different from the first distance.
[0196] In a further embodiment, the target data calculator is configured to modify or combine the frequency dependent variance or covariance parameters as rendering data items to obtain an overall combined variance or overall combined covariance parameter as an overall combination result, and is configured to calculate at least one of the interaural or interchannel coherence cue, the interaural or interchannel level difference cue, the interaural or interchannel phase difference cue, the first side gain, or the second side gain as target rendering data.
[0197] In a further embodiment, an initializer is provided for determining at least one of a left variance data item, a right variance data item, and a covariance data item from pre-stored head function data, the initializer being configured to calculate the left variance data item, the right variance data item, or the covariance data item from a plurality of head function data for a restricted spatial sector, the restricted spatial sector being sized such that at least two left head function data and at least two right head function data exist for the restricted spatial range.
[0198] References Alary, B., Politis, A., & Valimaki, V. (2017). Velvet Noise Decorrelator.
[0199] Baumgarte, F., & Faller, C. (2003). Binaural Cue Coding-Part I: Psychoacoustic Fundamentals and Design Principles. Speech and Audio Processing, IEEE Transactions on, 11(6), S.509-519.
[0200] Blauert, J. (2001).Spatial hearing(3 Ausg.).Cambridge;Mass:MIT Press.
[0201] Faller, C., & Baumgarte, F. (2003). Binaural Cue Coding-Part II: Schemes and Applications. Speech and Audio Processing, IEEE Transactions on, 11(6), S.520-531.
[0202] Kendall,G.S.(1995).The Decorrelation of Audio Signals and Its Impact on Spatial Imagery.Computer Music Journal,19(4),S.p 71-87.
[0203] Lauridsen,H.(1954).Experiments Concerning Different Kinds of Room-Acoustics Recording.Ingenioren,47.
[0204] Pihlajamaki,T.,Santala,O.,&Pulkki,V.(2014).Synthesis of Spatially Extended Virtual Source with Time-Frequency Decomposition of Mono Signals.Journal of the Audio Engineering Society,62(7 / 8),S.467-484.
[0205] Potard,G.(2003).A study on sound source apparent shape and wideness.
[0206] Potard,G.,&Burnett,I.(2004).Decorrelation Techniques for the Rendering of Apparent Sound Source Width in 3D Audio Displays.
[0207] Pulkki,V.(1997).Virtual Sound Source Positioning Using Vector Base Amplitude Panning.Journal of the Audio Engineering Society,45(6),S.456-466.
[0208] Pulkki,V.(1999).Uniform spreading of amplitude panned virtual sources.
[0209] Pulkki,V.(2007).Spatial Sound Reproduction with Directional Audio Coding.J.Audio Eng.Soc,55(6),S.503-516.
[0210] Pulkki,V.,Laitinen,M.-V.,&Erkut,C.(2009).Efficient Spatial Sound Synthesis for Virtual Worlds.
[0211] Schlecht,S.J.,Alary,B.,Valimaki,V.,&Habets,E.A.(2018).Optimized Velvet-Noise Decorrelator.
[0212] Schmele,T.,&Sayin,U.(2018).Controlling the Apparent Source Size in Ambisonics Unisng Decorrelation Filters.
[0213] Schmidt,J.,&Schroder,E.F.(2004).New and Advanced Features for Audio Presentation in the MPEG-4 Standard.
[0214] Verron,C.,Aramaki,M.,Kronland-Martinet,R.,&Pallone,G.(2010).A 3-D Immersive Synthesizer for Environmental Sounds.Audio,Speech,and Language Processing,IEEE Transactions on,title=A Backward-Compatible Multichannel Audio Codec,18(6),S.1550-1561.
[0215] Zotter,F.,&Frank,M.(2013).Efficient Phantom Source Widening.Archives of Acoustics,38(1),S.27-37.
[0216] Zotter,F.,Frank,M.,Kronlachner,M.,&Choi,J.-W.(2014).Efficient Phantom Source Widening and Diffuseness in Ambisonics.
Claims
1. An apparatus for synthesizing a spatially extended sound source (SESS) (7000), comprising: a storage device (200, 2000) for storing rendering data items for different elementary spatial sectors covering a rendering range for a listener; a sector identification processor (4000) for identifying from said different basic spatial sectors a set of basic spatial sectors belonging to said spatially extended sound source based on listener data and data of the spatially extended sound source; a target data calculator (5000) for calculating target rendering data from said rendering data items for said set of base spatial sectors; an audio processor (300, 3000) for processing an audio signal representative of the spatially extended sound source using the target rendering data; An apparatus comprising:
2. the storage device (200, 2000) is configured to store (810) as the rendering data items, for each basic spatial sector, at least one of a left variance data item relating to left head-related transfer function data, a right variance data item relating to right head-related transfer function (HRTF) data, and a covariance data item relating to the left HRTF data and the right HRTF data, the target calculator (5000) is configured to sum (830) the left variance data items of the set of base spatial sectors or the right variance data items of the set of base spatial sectors or the covariance data items of the set of base spatial sectors, respectively, to obtain at least one summed item; The target calculator (5000) is configured to calculate (840) at least one rendering queue as the target rendering data from the at least one summed item; the audio processor (300, 3000) is configured to process (850) the audio signal using the at least one rendering queue; 2. The apparatus of claim 1.
3. the sector identification processor (4000) is configured to apply a projection algorithm or a ray tracing analysis to determine the set of basic spatial sectors; configured to use listener position or listener orientation as the listener data, or to use information on the SESS orientation, the SESS position, or the geometry of the SESS as the spatially extended sound source (SESS) data; 3. Apparatus according to claim 1 or 2.
4. The sector identification processor (4000): receiving occlusion information for a potentially occluding object (7010) from a description of the audio scene; determining a particular spatial sector of the set of basic spatial sectors as an occluded sector based on the occlusion information; The device is configured to: the target data calculator (5000) is configured to apply an occlusion function to the stored rendering data items for the occluded sectors to obtain modified data (5020) and to use the modified data to calculate (5060) the target rendering data.
4. Apparatus according to any one of claims 1 to 3.
5. the occlusion function is a low pass function having different attenuation values for different frequencies, and the rendering data items are data items of different frequencies; the target data calculator (5000) is configured to weight (5020) for a number of frequencies a data item of a particular frequency with the attenuation value of the particular frequency to obtain the modified rendering data; 5. The apparatus of claim 4.
6. the sector identification processor (4000) is configured to determine (4010) that another basic spatial sector of the set of basic spatial sectors determined for the occluding object is not occluded by the potentially occluding object; the target data calculator (5000) is configured to combine (5040) the modified data from the occluded sector with the rendering data items of the other sectors, either without modification with the occlusion function or modified by a different modification function, to obtain the target rendering data.
6. Apparatus according to claim 4 or 5.
7. the sector identification processor (4000) is configured to determine a first basic spatial sector of the set of basic spatial sectors as having a first characteristic and to determine a second basic spatial sector of the set of basic spatial sectors as having a second, different characteristic; the target data calculator (5000) is configured to either not apply (4010) any modification function to the first basic spatial sector and to apply (4020) a modification function to the second basic spatial sector, or to apply (4020) a first modification function to the first basic spatial sector and to apply (4030) a second modification function to the second basic spatial sector, the second modification function being different from the first modification function; 7. Apparatus according to any one of claims 1 to 6.
8. 8. The apparatus of claim 7, the first modification function is frequency selective and the second modification function is constant over frequency, or the first modification function has a first frequency selective characteristic and the second modification function has a second frequency selective characteristic that is different from the first frequency selective characteristic, or the first modification function has a first attenuation characteristic and the second modification function has a second different attenuation characteristic; the target data calculator (5000) is configured to select or adjust the modification function from the first modification function and the second modification function based on a distance between the first basic spatial sector or the second basic spatial sector and the listener, or based on a characteristic of an object located between the listener and the corresponding basic spatial sector, Device.
9. the sector identification processor (4000) is configured to classify the set of base spatial sectors into different sector classes based on characteristics associated with the base spatial sectors; the target data calculator (5000) is adapted to combine (5020) the rendering data items of the basic space sectors of each class to obtain a combined result for each class if more than one basic space sector is in a class, and to apply a specific correction function associated with at least one class to the combined result for this class to obtain a modified combined result for this class, or applying a specific modification function associated with at least one class to said one or more data items of said one or more basic space sectors of each class to obtain modified data items, and combining said modified data items of said basic space sectors of each class to obtain a modified combined result of this class; Combining (5040) the combined results, or the modified combined results for each class, if available, to obtain an overall combined result; using the overall combined result as the target rendering data or calculating the target rendering data from the overall combined result (5060); 9. Apparatus according to any one of claims 1 to 8.
10. 10. The apparatus of claim 9, the characteristics of the basic spatial sectors are determined to be one of a group including a masked basic spatial sector including a first masking characteristic, a masked basic spatial sector including a second masking characteristic different from the first masking characteristic, an unmasked basic spatial sector having a first distance to the listener, and an unmasked basic spatial sector having a second distance to the listener, the second distance being different from the first distance; Device.
11. the target data calculator (5000) is adapted to modify or combine (5020, 5040) frequency dependent variance or covariance parameters as the rendering data items to obtain as the overall combination result an overall combined variance or overall combined covariance parameter, Calculating 5060 at least one of an interaural coherence cue, an interaural level difference cue, an interaural phase difference cue, a first side gain, or a second side gain as the target rendering data; 11. Apparatus according to claim 9 or 10.
12. the audio processor (300, 3000) is configured to perform at least one of an inter-channel coherence adjustment (320, 3200), an inter-channel phase difference adjustment (330, 3300), and an inter-channel level difference adjustment (340, 3400) using the corresponding cue as the target rendering data; 12. Apparatus according to any one of claims 1 to 11.
13. the rendering range comprises a sphere or a portion of a sphere around the listener, the rendering range being associated with the listener's position or listener's orientation, each elementary spatial sector having an azimuth and an elevation size; 13. Apparatus according to any one of claims 1 to 12.
14. the azimuth size and the elevation size of the basic spatial sectors are different from each other, whereby the azimuth size is finer for a basic spatial sector that is in front of the listener compared to the azimuth size of a basic spatial sector that is more to the side of the listener, or the azimuth size decreases towards the side of the listener, or the elevation size of a basic spatial sector is smaller than the azimuth size of this sector, 14. The apparatus of claim 13.
15. 1. A method for synthesizing a spatially extended sound source (SESS), comprising: storing rendering data items for different elementary spatial sectors covering a rendering range for a listener; - identifying from said different basic spatial sectors a set of basic spatial sectors belonging to said spatially extended sound source based on listener data and on data of the spatially extended sound source; calculating target rendering data from said rendering data items for said set of base spatial sectors; processing an audio signal representative of the spatially extended sound source using the target rendering data; A method comprising:
16. A computer program for carrying out the method of synthesis according to claim 15 when executed on a computer or processor.
Citation Information
Patent Citations
Acoustic processing apparatus and program
JP2017175356A
Dual listener positioning for mixed reality
JP2021514141A
Signaling spatial audio parameters
JP2021525392A
Apparatus and method for reproducing a spatially extended sound source or apparatus and method for generating a description for a spatially extended sound source using anchoring information
WO2021144308A1
Apparatus and method for synthesizing a spatially extended sound source using cue information items
WO2021180935A1
