Distributed interactive binaural rendering

Abstract

The present disclosure relates to a method, system, and computer program product for processing audio. A method includes receiving at least one input audio signal and generating a primary rendered presentation and an additional rendered presentation, each rendered presentation associated with a listener orientation and / or location. The method further includes determining transformation parameters for transforming the primary rendered presentation to an additional rendered presentation, and determining a deviation value based on the orientation and / or position of the user and the listener orientation and / or position. The method further includes determining a modified transformation parameter based on the transformation parameter and the deviation value, and applying the modified transformation parameter to the primary rendered presentation to generate an output presentation associated with the orientation and / or position of the user.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 63 / 340,181, filed on May 10, 2022, which is incorporated herein by reference in its entirety. Technical Field

[0003] The invention relates to a method for distributed rendering of audio signals. Background Art

[0004] Binaural audio content (e.g., in the form of stereo audio signals intended for playback on headphones or on a loudspeaker system with crosstalk cancellation) is becoming increasingly popular. For example, object-based audio content can be rendered as a binaural stereo presentation for headphones using a head-related transfer function (HRTF). Object-based audio content includes one or more audio objects associated with a position (optionally a time-varying position) in three-dimensional space. For example, an audio object may be intended to be perceived by a listener as an audio object moving to the right of the listener, above the listener, or along a trajectory around the listener. Thus, object-based audio can provide listeners with acoustic effects that enhance immersion.

[0005] HRTFs have been developed that describe the interaural time differences, interaural level differences, reflections occurring in the human ear, and the frequency response of the human ear as a function of the orientation and / or position of the listener's head. Using such HRTFs, binaural audio signals can be generated for arbitrary static or dynamic arrangements of audio objects in three-dimensional space. In addition, room reflections and / or reverberation are often added to create a sense of perceived distance and space.

[0006] In some cases, the rendering of object-based audio content is adjusted substantially in real time based on the orientation and / or position of the listener so that the audio object is fixed to the environment rather than to the listener's head. Therefore, when the listener moves his / her head, the rendering is adjusted so that the acoustic image is correspondingly displaced, so that the listener perceives the audio object as fixed in space rather than to his / her head. As an example, the listener is first presented with an audio rendering that renders the audio object as being perceived to be located on the right side of the listener. If the listener turns and faces the opposite direction, this orientation change is recorded by an orientation detector, which in turn provides this information to a renderer, which modifies the rendering to provide a modified rendering that renders the audio object as being perceived to be located on the left side of the listener. The effect of this is that the audio object is like being fixed in the listener's environment, wherein the listener can move and / or reorient himself / herself in this space. This form of orientation and / or position modified rendering (sometimes referred to as interactive binaural rendering) is particularly useful in gaming applications, extended reality (XR) applications, augmented reality (AR) applications, and virtual reality (VR) applications. Summary of the invention

[0007] A drawback of existing solutions for substantially real-time audio rendering based on listener orientation and / or position is that the rendering places high demands on data transmission bandwidth and processing power, which in turn increases the power consumption of the device performing the rendering. At the same time, in order to enable the rendering of convincing audio images, including audio objects that appear to be fixed in space or moving along trajectories fixed in space rather than fixed to the listener's head, it is important to keep latency (i.e., the time delay between a listener changing head orientation and / or position and the associated modification in the audio presentation) very low, typically on the order of tens of milliseconds.

[0008] Therefore, the first challenge is to provide an orientation and / or position based rendering process that provides sufficiently low latency and responds quickly to any changes in the listener's orientation and / or position. The latency between a change in orientation and / or position and the presentation of the modified audio presentation to the listener should ideally be much less than 100ms, as latency on the order of 17ms may be noticeable to many listeners. However, such low latency is difficult to achieve in practice due to the inherent latency introduced by the rendering process itself and the (usually wireless) transmission of sensor and audio data from an orientation tracking device worn by the user and a system, service or computer configured to perform the audio rendering.

[0009] To reduce latency, the orientation and / or position tracking device, audio renderer, and loudspeaker can be integrated into the same wearable device (e.g., earbuds or VR headset). However, a second challenge then arises, which is related to the computing power required for substantially real-time orientation / position-based rendering that responds quickly to changes in the listener's orientation and / or position, and the associated high power consumption. Object-based audio can include a large number of assets representing the environment, point sound sources, sound effects, dialogue, and other important elements, all of which need to be rendered in real time in response to changes in the listener's orientation and / or position, which may occur suddenly and very quickly (e.g., due to the listener quickly turning, looking up and down, or walking around in the environment). Wearable devices such as VR headsets, smart glasses, earbuds, or glasses typically do not have the required processing power, nor the battery capacity to maintain such audio rendering for a long time. Therefore, in many applications, orientation and / or location information is transmitted from the wearable device to a more powerful companion device, such as a phone, tablet, computer, game console, or cloud computer (e.g., edge server), which performs the rendering and transmits the rendered presentation back to the wearable device. However, communication between the companion device and the wearable device greatly increases latency, especially if the communication occurs over a public wireless communication channel such as Bluetooth, which may introduce significant latency.

[0010] To achieve sufficiently low latency, more powerful wearable devices with enhanced processing performance and, for example, larger batteries can be used. However, in order to physically accommodate the enhanced device capabilities, a third challenge then arises, as the wearable device becomes bulky and inconvenient to use (e.g., larger and / or heavier to accommodate the necessary processing components, power components, and cooling components). Typically, the bandwidth used to communicate with the wearable device is also limited, and because multiple audio elements in object-based audio content require considerable bandwidth, some audio elements may need to be removed or compressed, which reduces the quality of experience (QoE). Due to the difficulty in obtaining sufficient bandwidth using wireless communications, some solutions resort to wired data connections to wearable devices, however, this greatly hinders the flexibility of wearable devices, making them difficult to use outdoors or for users to move freely.

[0011] An object of the present disclosure is to propose a method for rendering audio content, in particular object-based audio content, which method responds substantially in real time to changes in listener orientation and / or position, thereby overcoming or at least alleviating the problems of existing solutions highlighted above.

[0012] According to a first aspect of the present invention, there is provided a method of processing audio, the method comprising: receiving at least one input audio signal at a first processing module, and generating at the first processing module a primary rendered presentation and an additional rendered presentation, each rendered presentation being associated with a first listener orientation and / or position and a second listener orientation and / or position, respectively. The method further comprises determining at the first processing module transformation parameters for transforming the primary rendered presentation into an additional rendered presentation, and receiving at the second processing module the transformation parameters and the primary rendered presentation generated by the first processing module. The method further comprises receiving at the second processing module user orientation and / or position data indicating an orientation and / or position of a user, determining at the second processing module an orientation and / or position deviation value based on the orientation and / or position of the user and the first listener orientation and / or position and the second listener orientation and / or position, determining at the second processing module modified transformation parameters based on the transformation parameters and the orientation and / or position deviation value, and applying at the second processing module the modified transformation parameters to the primary rendered presentation to generate an output presentation associated with the orientation and / or position of the user.

[0013] That is, the first processing module pre-renders at least two presentations associated with different listener orientations and / or positions and determines for each presentation except one (the primary presentation) transformation parameters that may be used to transform the primary presentation into at least one additional rendered presentation.

[0014] A listener or user "orientation" refers to the assumed rotational orientation of the listener's or user's head. For example, the orientation may be defined by one or more of a pitch angle, a yaw angle, and a roll angle. A listener or user "position" refers to the position of the listener's or user's head in one or more of the front / back, left / right, and up / down directions. For example, the position may be defined by a Cartesian coordinate system having vertical X-, Y-, and Z-axes. It should be understood that different listener orientations and / or positions may differ in one of the orientation and position, or in both the orientation and position. It is contemplated that some embodiments only consider orientation changes (with one, two, or three degrees of freedom), while in other embodiments only consider position changes (with one, two, or three degrees of freedom).

[0015] The orientation and / or position deviation value can be a linear or non-linear distance between two orientations and / or positions. In addition, the orientation and / or position deviation can be a perceptually weighted distance between two orientations and / or positions, as will be described in further detail below.

[0016] The transformation parameters may be updated for each time-frequency slice of the time-frequency representation. As will be described below, for an audio presentation with two channels, each transformation parameter set may include as few as four or five transformation parameters (some of which may be complex-valued) or even as few as two real-valued transformation parameters, which constitutes an amount of data that may be transmitted quickly with low latency. The transformation parameters are still sufficient to accurately describe the orientation / position transformation from the primary presentation to the additional presentation, and may be used to find modified transformation parameters (using, for example, interpolation) in the event that the user orientation / position does not correspond to the orientation / position associated with the additional presentation.

[0017] Hence, even if the transform parameters are updated frequently, e.g. for each time-frequency tile, they represent only a small amount of data (compared to the hundreds or thousands of samples used to represent the time-frequency tiles of an audio channel), which can be efficiently transmitted to the second processing module.

[0018] Furthermore, the application and / or modification of the transformation parameters is computationally efficient and can be performed quickly even on processing modules with limited processing power, which means that the second processing module can be implemented on limited devices such as headphones, earbuds, wireless earbuds, true wireless earbuds, smart glasses, or VR / AR / XR headsets. By receiving a rendered primary presentation associated with a first listener orientation / position and transformation parameters associated with a second listener orientation / position, the second processing module can quickly modify the transformation parameters and apply them to the primary presentation to shift the presentation to the second listener orientation / position if this is more consistent with the actual user orientation / position. The transformation parameters can also be modified, for example using interpolation, before applying the transformation parameters to the primary presentation to more accurately follow the user's orientation / position.

[0019] In this way, the rendering of the input audio signal can be shifted based on the user's orientation / position, so that the user is presented with an audio presentation that is fixed in space. As an illustrative example, the audio assets are associated with music from a virtual stage directly in front of the listener, and the user listens to these audio assets using earbud headphones while standing in the physical space. If the user turns his or her head to the right, the rendering is adjusted so that the listener is presented with an audio presentation as if the music is coming from the left. This is an example of modifying the presentation to follow the user's orientation relative to the virtual three-dimensional space of the audio asset. If the listener moves toward or away from the virtual stage, the user can be presented with an audio presentation in which the music becomes louder or weaker. This is an example of modifying the presentation to follow the user's position relative to the virtual three-dimensional space of the audio asset. One or more audio assets may also include an audio object that moves along a track in the virtual three-dimensional space. By shifting the rendering of the audio asset based on the user's orientation / position, the listener can be provided with an audio presentation that makes the listener perceive that the track along which the audio object moves is fixed in the virtual three-dimensional space.

[0020] In some embodiments, the first listener orientation and / or position is a different yaw orientation from the second listener orientation and / or position in the corresponding first pitch orientation and second pitch orientation, and the method further includes obtaining, at the second processing module, simplified transformation parameters associated with a third pitch orientation, the simplified transformation parameters being configured to transform the primary rendered presentation or the additional rendered presentation into a pitch rendered presentation having the third pitch orientation, and applying the simplified transformation parameters to the primary rendered presentation based on the orientation deviation at the second processing module to generate an output presentation.

[0021] That is, each set of transformation parameters may be associated with a respective orientation that is different in yaw (user looking up or down) at a predetermined pitch angle (user looking left or right), and the transformation parameters capture the interaural effects that are very evident for varying yaw angles. On the other hand, for each yaw angle, in order to span different pitch angles, a simplified set of transformation parameters having fewer parameter values ​​(e.g. one actual gain value per channel) compared to the (unsimplified) transformation parameters is transmitted for multiple pitch angles that deviate from the predetermined pitch angle. Thus, by taking into account that the sensitivity to audio presentation shifts in yaw is different from the sensitivity to presentation shifts in pitch, the amount of information transmitted to the second processing module may be reduced without reducing the quality of experience QoE.

[0022] According to a second aspect of the present invention, there is provided a computer program product, the computer program product comprising instructions which, when the program is executed by a computer, cause the computer to perform the method according to the first aspect.

[0023] According to a third aspect of the present invention, there is provided a system comprising a first processing module in communication with a second processing module, wherein the first processing module and the second processing module are configured to perform the method according to the first aspect.

[0024] The computer program product according to the second aspect and the system according to the third aspect are characterized by the same or equivalent benefits as the method according to the first aspect. Any functionality described with respect to the method may have corresponding features in the system or computer program product and vice versa. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Aspects of the invention will be described in more detail with reference to the accompanying drawings, which show currently preferred embodiments.

[0026] Figure 1 A distributed audio processing system according to some implementations is shown.

[0027] Figure 2 Schematically illustrates a plurality of distributed listener orientations / positions according to some embodiments.

[0028] Figure 3 Schematically illustrates a plurality of distributed listener orientations / positions and detected user orientations / positions that are different from two of the listener orientations / positions along the X-axis or the yaw axis according to some embodiments.

[0029] Figure 4 Schematically illustrates a plurality of distributed listener orientations / positions and detected user positions / orientations that are different from two of the listener orientations / positions along the Y-axis or the pitch axis according to some embodiments.

[0030] Figure 5 Schematically illustrated are a plurality of distributed listener orientations / positions and two detected user orientations / positions according to some embodiments.

[0031] Figure 6 Distributed listener orientations / positions are schematically illustrated according to some embodiments, wherein separate listener orientations in yaw are associated with transformation parameters, and wherein for each yaw orientation axis, multiple listener pitch orientations are associated with simplified transformation parameters.

[0032] Figure 7 A multi-representation encoder communicating with an interactive renderer using a parameter encoder and a parameter decoder is shown according to some embodiments.

[0033] Figure 8 is a flow chart describing a method for processing audio according to some implementations.

[0034] Fig. 9 is a flow chart describing a process of determining transformation parameters and simplifying the transformation parameters according to some implementations.

[0035] Fig.10 is a flow chart describing a process of transmitting transformation parameters to an interactive renderer according to some implementations. DETAILED DESCRIPTION

[0036] The systems and methods disclosed in this application can be implemented as software, firmware, hardware, or a combination thereof. In hardware implementations, the task division does not necessarily correspond to the division of physical units; on the contrary, a physical component can have multiple functions, and a task can be performed by several physical components in collaboration.

[0037] The computer hardware may be, for example, a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular phone, a smart phone, an AR / VR wearable automotive infotainment system, a web device, a network router, a switch or a bridge, or any machine capable of (sequentially or otherwise) executing instructions specifying actions to be taken by the computer hardware. Further, the present disclosure will relate to any collection of computer hardware that executes instructions, alone or in combination, to perform any one or more of the concepts discussed herein.

[0038] Some or all of the components may be implemented by one or more processors that accept a computer-readable (also referred to as machine-readable) code containing a set of instructions that, when executed by one or more processors, perform at least one method described herein. Any processor that is capable of executing (sequentially or otherwise) a set of instructions that specifies an action to be taken is included. Thus, an example is a typical processing system (e.g., computer hardware) that includes one or more processors. Each processor may include one or more of a CPU, a graphics processing unit, and a programmable DSP unit. The processing system may further include a memory subsystem that includes a hard drive, an SSD, a RAM, and / or a ROM. A bus subsystem may be included for communication between components. The software may reside in the memory subsystem and / or the processor during its execution by the computer system.

[0039] The one or more processors may operate as standalone devices or may be connected to other processor(s), such as via a network. Such a network may be constructed on a variety of different network protocols and may be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof.

[0040] The software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transient medium). As is well known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include, but are not limited to, various forms of physical (non-transitory) storage media, such as EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage devices, cassettes, tapes, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Further, it is well known to those skilled in the art that (transitory) communication media typically embodies computer-readable instructions, data structures, program modules or other data in the form of modulated data signals or other transmission mechanisms such as carrier waves, and includes any information transfer medium.

[0041] Figure 1 A distributed rendering system 1 according to some embodiments is depicted. The distributed rendering system 1 comprises three subsystems 11, 13, 15. More specifically, the distributed rendering system 1 comprises a multi-rendering renderer module 11, a multi-rendering encoder 13, and an interactive renderer 15. At least one of the subsystems 11, 13, 15 is implemented in a device separate from a device implementing at least one of the other subsystems 11, 13, 15, which means that the entire rendering process is distributed over at least two devices that communicate with each other. Two of the subsystems 11, 13, 15 may be implemented in the same device, wherein the remaining subsystems 11, 13, 15 are implemented by separate devices.

[0042] As will be described below, the amount of input data and the computational complexity of the processing performed vary between the different subsystems. Figure 1 The benefit of the distributed rendering system 1 is that the amount of data transmitted to the interactive renderer 15 is minimized, while the interactive renderer 15 is also associated with the least complex processing of the three subsystems. This makes the interactive renderer 15 very suitable for implementation in computationally limited and power-constrained devices (such as wearable devices), while the other two subsystems 11, 13 can be implemented in more computationally powerful devices (such as smartphones, computers or game consoles that communicate with the device implementing the interactive renderer 15).

[0043] Thus, in some embodiments, the multi-rendering renderer 11 and the multi-rendering encoder 13 are implemented on a high-performance device (or optionally on two different high-performance devices that communicate with each other), while the interactive renderer 15 is implemented on a separate restricted device, wherein the high-performance device is configured to communicate with the restricted device. Examples of high-performance devices may be smartphones, tablet computers, computers (e.g., desktop computers or laptop computers), game consoles, cloud computers, or servers. Examples of restricted devices include a pair of headphones, earbuds, wireless earbuds, smart glasses, true wireless earbuds, or VR / AR / XR head-mounted devices. It may be beneficial for a restricted device to communicate with a high-performance device using a wireless connection (e.g., WiFi or Bluetooth), although it is also contemplated that communication may also be performed over a wired connection.

[0044] Now refer to Figure 1 The processing performed by the multi-rendering renderer 11, the multi-rendering encoder 13, and the interactive renderer 15 is described in more detail.

[0045] The multi-rendering renderer 11 is configured to render at least two audio renderings based on one or more audio assets 10. The audio renderings are labeled R1, ..., R p , …, R P , which means that the multi-rendering renderer 11 usually renders P number of renderings, where P ≥ 2. The at least two renderings R1, ..., R P Each of is associated with a different listener orientation and / or position relative to the audio asset 10. The term "listener orientation and / or position" is used to refer to an assumed listener orientation / position relative to the audio asset 11.

[0046] The audio asset 10 may include one or more spatialized audio objects, often referred to simply as audio objects. An audio object is an audio signal associated with a spatial property such as a position or direction of incidence in three-dimensional space. How one or more audio objects should be rendered to form an audio presentation depends on the orientation and / or position of the assumed listener relative to the audio objects.

[0047] The multi-rendering renderer 11 selects the audio assets 10 labeled V1, V2, ..., V P , and for the plurality of listener orientations / positions V1, V2, ..., V P Each of the renderings in R1, ..., R P For example, the plurality of listener orientations / positions V1, V2, ..., V Pis selected to span a range of orientations (indicated by pitch, yaw and roll angles) and / or positions (indicated by Cartesian coordinates X, Y, Z) in the three-dimensional space of the audio asset. It should be noted that the multi-rendering renderer 11 may select the listener orientation / position without regard to any actual measured orientation / position of the user. That is, the multi-rendering renderer 11 renders multiple possible renderings that will be associated with the audio asset in the order V1, V2, ..., V P However, these positions usually do not correspond to the actual user orientation V L Completely corresponds.

[0048] For example, each audio presents R1, ..., R P is a pair of binaural audio signals extracted using corresponding HRTFs, wherein the orientation / position of the HRTFs relative to the audio asset 10 is different between the corresponding HRTFs.

[0049] That is, the multi-rendering renderer 11 obtains at least two orientations V1, ...V p ,…,V P , and renders corresponding renderings R1, ...Rp, ...R for each orientation based on the audio asset 10 P In some embodiments, Figure 2 Schematically shown in FIG. 1 , orientation / position V1, ... V p ,…,V P Spanning different combinations of pitch and yaw angles at predetermined points in the three-dimensional space of the audio asset 10. For example, orientation V1 indicates a pitch of 0 degrees and a yaw of 0 degrees, orientation V2 indicates a pitch of 0 degrees and a yaw of 5 degrees, orientation V3 indicates a yaw of 5 degrees and a yaw of -5 degrees, etc. Similarly, orientations / positions may be selected to span various X, Y, Z positions.

[0050] For each orientation / position V1, ..., V P , render separate audio representations R1, ..., R P To achieve this, the multi-rendering renderer 11 may include a plurality of renderers 12a, 12b, 12c, each renderer being associated with a single orientation / position V1, ..., V P associated and configured based on orientation / position V1, ..., V P and audio asset 10 to render the associated renderings R1, ..., R P In some embodiments, each presents R1, ..., R P It is a binaural audio presentation suitable for playback on headphones, the headphones comprising two audio channels, i.e. a left audio channel and a right audio channel. However, it is contemplated that the presentation may also be other types of presentations, such as a mono presentation, a stereo presentation, or a surround presentation (e.g., a 5.1 or 7.1 presentation).

[0051] The multi-rendering renderer 11 renders at least two representations R1 and R2. Generally, it is beneficial if the multi-rendering renderer 11 renders a large number of representations, such as at least ten representations (P≥10), at least twenty representations (P≥20), or at least fifty representations (P≥50) to span a large area of ​​listener orientations / positions and / or to ensure that the distance between two listener orientations / positions is not too large.

[0052] The multi-rendering renderer 11 renders the rendered presentations R1, ..., R P Transmitted to the multi-presentation encoder 13.

[0053] The multi-rendering encoder 13 receives all P representations R1, ..., R from the multi-rendering renderer 11. P , and determine the set of transformation parameters W for all but one rendering p That is, the multi-rendering encoder 13 designates one of the P representations as the primary representation and determines associated transform parameters for all (at least one) remaining representations. The remaining (multiple) representations are referred to as additional representations. In the following, and without loss of generality, it is assumed that representation R1 is the primary representation, which means that representations R2, ..., R P is the additional presentation of R2, ..., R P , and present R2, …, R for the rest P Each of the transform parameters W2, ..., W P .

[0054] Each transformation parameter set W p is configured to transform the primary presentation R1 at the listener orientation V1 to a primary presentation R2 at the listener orientation V p Presentation R p , where the index p ranges from 2 to P, where P≥2.

[0055] In order to determine the transformation parameter W p The multi-rendering encoder 13 comprises one or more parameter generators 14b, 14c, wherein each parameter generator 14b, 14c takes two renderings as input, namely a main rendering R1 and an additional rendering R2, ... R P Each parameter generator 14b, 14c generates a parameter that transforms the main presentation R1 into a corresponding additional presentation R P The transformation parameter W P .

[0056] In the following, the operation of parameter generator 14b and the nature of the transformation parameters will be described in detail. It should be understood that other types of transformation parameters may also be similarly determined and used, and any other parameter generator 14a may operate completely similarly to parameter generator 14b.

[0057] As mentioned above, the parameter generator 14b receives two rendered presentations, namely a main presentation denoted R1 and an additional rendered presentation R2. p These two present R1, R p The format is the same, for example, the main presentation R1 and the additional rendered presentation R p are binaural audio signals, stereo audio signals, mono audio signals or surround audio signals (eg 5.1 signals). In the following exemplary embodiments, it will be assumed that R1, R p The format of is a binaural format comprising two audio channels, however it should be noted that the same process can be similarly performed for presentation of other formats.

[0058] Each rendered image R1, R p Including left and right channels (forming a binaural signal pair). Therefore, for the main presentation R1 and the additional presentation R p , which maintains

[0059]

[0060]

[0061] in, represents the left channel of the pth presentation, and represents the right channel of the pth presentation. In some embodiments, the parameter generator 14b determines the transformation matrix Make

[0062]

[0063] Among them, [z l，p [n],z l，p [n]] T is in orientation / position V p Reconstruction of And the index n indicates the audio sample index of the corresponding channel. Therefore, Used transformation matrix Reconstructed from R1 in orientation / position V1, where ideally

[0064] To determine the transformation matrix The parameter generator can be rendered by minimizing R p With the matrix applied The initial least squares solution is determined by the root mean square error between the main presentation R1 That is, the initial least squares solution can be determined as

[0065]

[0066] The initial least squares solution from Equation 4 can be determined iteratively Alternatively, the initial least squares solution can be expressed as a closed-form solution. For example, if the covariance matrix of the presented channels (e.g., ) is represented by R y，p，p , and two different presentations R p1 and R p2 The correlation matrix between them is represented as R y，p1，p2 , it can be shown that the closed-form solution of Equation 6 is given by:

[0067]

[0068] where ∈ is the regularization parameter and I is the identity matrix. If the transformation matrix Directly applied to the main presentation R1, then the presentation R p Approximate value of As follows

[0069]

[0070] in, Indicates the presentation of R p Approximate value of The transformation matrix Can be used as the transformation parameter W P , because it transforms the main presentation R1 into presentation R p Therefore, in some embodiments, the transformation parameters include the transformation matrix Among them, the transformation matrix The size is N times N, where N is the size of the main rendered representation R1 and the size of the additional rendered representation R p the number of audio channels in the presentation format, and Instructions are similar to those for attaching R p The main presentation R1 audio channel combination. It should be noted that the transformation matrix M may have complex-valued elements and / or be a three-dimensional matrix of size N×N×M, where M is a number of real or complex filter samples.

[0071] In some embodiments, it is desirable to present additional pA more accurate reconstruction is performed, thereby determining the additional diagonal gain matrix G and / or the decorrelation contribution gain g d,p and use it to modify the transformation matrix To obtain the enhanced modified transformation matrix M p For example, even if Solving Equation 4, applying it to the primary rendering R1 may, in some cases, produce the same p The covariance matrix of is different from the covariance matrix of . The predictions are presented The covariance matrix of Given, it can be expressed as

[0072]

[0073] in, may be different from the actual presentation R p The covariance matrix R y，p，p This deviation in the covariance matrix can be obtained by the diagonal gain matrix G and / or by the decorrelation gain g d,p Specifically, for the main presentation R1 The enhanced reconstruction model is established as

[0074]

[0075] in,

[0076]

[0077]

[0078] and

[0079]

[0080] Here, the function Ψ(·) represents a decorrelator that generates the input signal from the output signal according to the following two requirements:

[0081] <Ψ(x) 2 >= <x 2 >, Equation 12

[0082] as well as

[0083] <Ψ(x)>=0 Equation 13

[0084] Here, <·> represents the expected value operator.

[0085] From Equation 8, we can see that assuming g d，p ≤0, the enhanced prediction The resulting covariance matrix R z，p，pGiven by the following formula

[0086]

[0087] in,

[0088]

[0089] Therefore, by setting the diagonal gain matrices G and g d,p The appropriate value of The covariance matrix R z，p，p Equal to true present R p The covariance matrix R y，p，p It can be seen that R z，p，p =R y，p，p The solution to this problem is obtained as follows:

[0090]

[0091]

[0092] and

[0093]

[0094] in,

[0095]

[0096] and

[0097]

[0098] Therefore, the enhanced transformation matrix M is obtained p As And for a single presentation R p , the parameter generator 14b determines (A) the transformation matrix Or (B) the enhanced transformation matrix M p and decorrelation gain g d,p As the transformation parameter W p This information is sufficient to accurately reconstruct the rendering R from the primary rendering R1. p .

[0099] In some embodiments, for presentation of R1 and R p Each time-frequency slice of is repeated to determine the transformation matrix Or the enhanced transformation matrix M p and decorrelation gain g d,p The parameter generator 14b determines an initial least squares solution for each time-frequency slice presented accordingly. Or the enhanced transformation matrix M pand decorrelation gain g d,p . Therefore, even if the representation varies with time and / or frequency, the initial least squares solution Or the enhanced transformation matrix M p and decorrelation gain g d,p It also enables accurate reconstruction of the rendering R from R1 p .

[0100] It should be noted that the transformation matrix Or the enhanced transformation matrix M p The elements of can be complex-valued. In addition, it should be noted that the transformation matrix Or the enhanced transformation matrix M p Each element of may be a vector value (eg, forming a three-dimensional matrix), wherein each element defines a plurality of discrete real-valued or complex-valued filter samples that define a FIR filter.

[0101] In the above example, R1, ...R p is a binaural audio signal with two channels. In such an example, and M p is a 2×2 matrix or (2×2×M matrix, where M is the number of discrete filter bank samples). In general, for a presentation with N channels, and M p is an N×N matrix or an N×N×M matrix.

[0102] For each representation and time-frequency slice, the transformation matrix Or the enhanced transformation matrix M p and decorrelation gain g d,p is combined into a transformation parameter set W p For example, for each presentation and time-frequency slice, W p ={M p , g d，p}or

[0103] Each rendered transformation parameter W p Together with the main rendering R1, it is transmitted to the interactive renderer 15. It should be understood that compared with the main rendering R1, the transformation parameter set W p Much less data needs to be transmitted. For example, the main presentation R1 is associated with a time-frequency slice representation (e.g., STFT representation) having hundreds or thousands of complex-valued audio samples in a frame. On the other hand, the presentation R p The associated transformation parameter W p Each time-frequency slice is presented in When only four (potentially complex) values ​​are involved, or when W p ={Mp , g d，p} involves five values ​​(four of which are potentially complex numbers). Assume that all STFT frequency audio samples in each frame will be divided into 5 to 50 slices or bands for which the transform parameters W are calculated p , then the transform parameters will include approximately 25 to 250 parameter values, which is one to two orders of magnitude smaller than the audio samples in the main presentation. In addition to the reduced number of parameters, the accuracy of the transform parameters will usually be significantly lower than that required for the audio samples, which also contributes to the reduction of information when the transmitted data is digitally quantized. Therefore, it is not necessary to transmit the additional presentation R p Compared to transmitting an additional set of transformation parameters W p More efficient.

[0104] Mainly presents R1 and V in the same orientation / position p At least one transformation parameter set W associated with p are received by the interactive renderer 15. Since it is not possible to p The associated orientation / position V is directly derived from p What is the orientation / position V p can be explicitly indicated as accompanying each transformation parameter set W p The vector V p Alternatively, the orientation / position V1, ..., V P It can be predetermined in advance and stored locally in the interactive renderer 15.

[0105] Further references Figure 2 , which schematically illustrates positions V2, ..., V P How to distribute across a range of translational positions and / or rotational orientations. Figure 2 The positions V1, V2, V3, V4 illustrated in the figure are distributed at different translation positions in the XY plane and / or different rotation orientations in the pitch-yaw plane. It should be understood that it is also possible to Figure 2 Additional orientations / positions are added in the third dimension (Z-axis or roll axis) of the XY plane or the pitch-yaw plane as shown in . Position V1 is associated with the primary presentation R1, and for the remaining positions V2, V3, V4, the associated transformation parameters W2, W3, W4 are available to the interactive renderer 15 to transform the primary presentation R1 into a presentation associated with one of the positions V2, V3, V4.

[0106] The interactive renderer 15 also receives a signal indicating the orientation / position V of the user. LThe listener orientation / position data may be received from an orientation / position detector, such as a head tracking detector. Examples of orientation / position detectors include magnetic sensors, gyroscopic sensors, GPS receivers, accelerometers, and UV / IR / visible light sensors (e.g., camera sensors). The orientation / position detector may be included in the same device as the interactive renderer 15, thereby enabling low-latency communication between the orientation detector and the interactive renderer 15. For example, the orientation detector and the interactive renderer 15 may be included in a pair of earbuds or earplugs. Additionally or alternatively, an orientation / position tracker is disposed outside of the device implementing the interactive renderer 15, wherein the orientation / position tracker transmits the user orientation and / or position data to the interactive renderer 15. An example of an external orientation tracking device is one or more cameras disposed in the user environment, the one or more cameras tracking the orientation / position of the user's head, for example using motion tracking or facial recognition. Another example of an external orientation tracker is a user wearing one or more light-emitting devices that emit IR, UV, or visible light that is tracked by one or more IR, UV, or visible light sensors set up in the environment, where the orientation / position of the light-emitting devices is associated with the orientation / position of the user's head.

[0107] like Figure 3 As shown, the user orientation / position V L may deviate from one or more of the orientations / positions V1, V2, V3, V4 for which the transformation parameters W2, W3, W4 have been generated. L The interactive renderer 15 may vary continuously or at a finer granularity than the orientation / position V1, V2, V3, V4. For example, the user has moved his or her head to face a yaw orientation different from the yaw orientation V1 of the primary presentation R1 and the yaw orientation V2 of the presentation R2 for which the transformation parameters are available. To this end, the interactive renderer 15 includes a parameter processor 17 configured to determine, based on the user's orientation / position V1, V2, V3, V4, the yaw orientation V2 of the primary presentation R1 and the yaw orientation V2 of the presentation R2 for which the transformation parameters are available. L The modified transformation parameter W is determined by the deviation value between at least one of the following items: L : orientation / position V1 associated with primary presentation R1 and orientation / position V2, V3, V4 associated with transformation parameter sets W2, W3, W4.

[0108] Determine the modified transformation parameters W L Can include selection and listener orientation / position V L The transformation parameters associated with the orientation / position V1, V2, V3, V4 of the smallest distance (eg, closest) are determined, and these parameters are used to transform the primary presentation R1. Alternatively, the modified transformation parameters W are determinedL can include the listener orientation / position V L Interpolation is performed between at least two transformation parameter sets W1, W2, W3, W4 associated with nearby orientations / positions V1, V2, V3, V4.

[0109] Between two listener orientations / positions V1, V2, V3, V4 and / or between a listener orientation / position V1, V2, V3, V4 and a user orientation V L The deviation value between can be determined as a linear or nonlinear distance between orientations / positions. For example, each orientation / position V1, V2, V3, V4, V L can be represented as a point or vector in a coordinate system, and two orientations / positions V1, V2, V3, V4, V L The deviation value can be simply determined as the Euclidean distance, cosine similarity, haversine distance, etc. between two points or vectors.

[0110] In addition, the orientations / positions V1, V2, V3, V4 of two listeners and / or the orientations / positions V1, V2, V3, V4 of the listeners and the orientations / positions V1, V2, V3, V4 of the user L The deviation value between them can be a perceptual weighted distance. That is, to determine the two orientations / positions V1, V2, V3, V4, V L The deviation values ​​therebetween may include weighting different components of distance (expressed in, for example, pitch, yaw, roll, and / or X, Y, Z distance) differently based on the expected perceptual impact on the rendered presentation.

[0111] For example, as will be explained in detail below, a change in orientation in yaw (i.e., the user looking left and right) may be more perceptually significant than a change in orientation in pitch (the user looking up and down). L When the deviation value is between , any yaw distance can be weighted more than the pitch distance, so that, for example, the user orientation V L When the closest listener orientations V1 , V2 , V3 , V4 are selected, preference is given to listener orientations V1 , V2 , V3 , V4 having similar pitch orientations rather than similar yaw orientations.

[0112] The same analogy applies when determining deviation values ​​for different positions, which may be represented, for example, in a Cartesian coordinate system having an X-direction, a Y-direction, and a Z-direction, wherein a change in position in one direction may have a greater perceived impact on the rendered presentation than a change in position in another direction. Thus, the distances along different axes may be weighted differently, such that the distance along one axis may have a greater impact on the deviation value than the distance along another axis.

[0113] Additionally, when determining deviation values, perceptual weighting may place greater emphasis on distance in orientation than distance in position, or vice versa. For example, in many embodiments, it may be beneficial to provide greater weight to distance in orientation than distance in position because changes in orientation may have a stronger perceptual effect than changes in position. As an example, if an audio asset includes an audio object located at a large distance from a user, a small change in position may have a negligible perceptual effect, while a small change in orientation may still have a larger perceptual effect. Therefore, when determining a deviation value between a user orientation / position and a listener orientation / position V1, V2, V3, V4, the deviation value will depend primarily on the difference in orientation.

[0114] As explained above, the modified transformation parameters W are determined L Including determining the user orientation V L a deviation value from at least one of: a listener orientation / position V2, V3, V4 associated with the transformation parameters W2, W3, W4 and an orientation / position V1 associated with the primary presentation R1, wherein the deviation value is a user orientation V L A measure of linear distance, non-linear distance and / or perceptually weighted distance to the listener orientations V1, V2, V3, V4.

[0115] therefore, Figures 2 to 6 The scaling of the axes in can be linear, nonlinear, and / or perceptually distorted. For example, Figure 2 The listener orientations / positions V1, V2, V3, V4 in may not be evenly distributed in linear space, because the linear distance in pitch between V1 and V3 may be much larger than the linear distance in yaw between V1 and V2, even though these distances (with twisted or non-linear axes) appear to be roughly the same.

[0116] refer to Figure 4 , we will now describe the determination of the listener orientation / position V L The transformation parameters associated with the orientation / position V1, V2, V3, V4 with the smallest distance (e.g., the closest). Lis closer to (i.e., associated with a smaller deviation value) orientation / position V2, then the transformation parameter W2 is set by W L =W2 is selected by the parameter processor 17. While determining and using the transformation parameters associated with the closest, best matching orientation / position V1, V2, V3, V4 is a process that can be performed very quickly and efficiently, there may be significant acoustic disturbances when the user changes orientation and selects different transformation parameters. On the other hand, if many transformation parameters have been generated for a fine-grained distribution of orientations / positions V1, V2, V3, V4, the significance of these acoustic disturbances can be mitigated.

[0117] Alternatively, the modified transformation parameters W are determined L may include user orientation / position V1, V2, V3, V4 based on the orientation / position V1, V2, V3, V4 L Interpolation is performed between the transformation parameters W2, W3, W4 associated with different orientations / positions V1, V2, V3, V4. Figure 4 As shown, the user orientation / position V L is between orientation / position V4 and V2, whereby the modified transformation parameter W is determined L Including V-based L The deviation between V2 and V L The deviation value from V4 is interpolated between parameter W4 associated with orientation / position V4 and parameter W2 associated with orientation / position V2.

[0118] Thus, the transformation parameters associated with the orientations / positions between the discrete orientations / positions V1, V2, V3, V4 can be accessed via interpolation (e.g., linear interpolation) or using a triangulation interpolation function. The transformation parameters W1, W2, W3, W4 for each position can all be in the same format. As described above, each transformation parameter set W p The initial least squares solution transformation matrix can be indicated Or every set W p The transformation matrix M that can be used to transform the enhancement of the main presentation can be indicated p and decorrelation gain G d,p .

[0119] The main presentation R1 is associated with an orientation / position V1. However, since there may be no need to determine any transformation parameters associated with this orientation / position, it is envisaged that the parameter processor 17 may associate the position V1 with default transformation parameters W1, for example W1 = {M p , g d，p}, where M p =1 and g d，p= 1, thereby allowing the orientation / position V1 and the associated default transformation parameters W1 to be used for interpolation. Figure 3 As shown, the user orientation / position V L can be located between V1 and V2, whereby the parameter processor 17 is based on V L The deviation between V1 and V L The deviation value from V1 performs an interpolation between the default parameter W1 associated with orientation / position V1 and the parameter W2 associated with orientation / position V2 to obtain W L .

[0120] exist Figure 2 and Figure 3 , user orientation / position V L is illustrated as being located along a DOF axis (e.g., along Figure 3 However, it should be understood that the user orientation / position V L can be any orientation / position in a 6-DOF system, which means that interpolation can be performed between more than two points. Figure 5 As shown, the user orientation / position V L can be located between, but separate from, multiple orientations / positions V1, V2, V3, V4 associated with different individual transformation parameters W1, W2, W3, W4, wherein based on the user orientation / position V L Deviation values ​​from the orientations / positions V1, V2, V3, V4 perform interpolation between the transformation parameters W1, W2, W3, W4.

[0121] The main presentation R1 and the transformation parameters W2, W3, W4 may also be transmitted to a second interactive renderer (not shown) implemented in another device separate from the device implementing the first interactive renderer 15. The second interactive renderer performs processing corresponding to the first interactive renderer 15, but based on the user orientation / position V of the second user. L2 That is, the second interactive renderer determines a second modified set of transformation parameters based on the same transformation parameters W2, W3, W4, which can be used to transform the same primary presentation R1 into a second user orientation / position V L2 In other words, the distributed rendering system 1 may be used for multicasting, where the same audio asset 10 is rendered to multiple users via respective interactive renderers 15 that obtain the orientation of the respective users, but using the same multi-rendering renderer 11 and multi-rendering encoder 13 .

[0122] After determining (by interpolation or selection of the best matching parameters) the modified transformation parameters W LThen, the parameter processor 17 transforms the parameter W L The rendering transformer 15 provides the modified transformation parameters W L Applied to the primary presentation R1 to generate an output presentation. The output presentation may be referred to as an interactive output presentation. Optionally, the output presentation is provided to one or more loudspeakers (e.g., loudspeakers of a pair of earphones or headphones) that present the output presentation to a user.

[0123] The transformation parameters W1, W2, W3, W4 for each (optionally partially overlapping) frame and band are updated regularly and provided to the interactive renderer 15. Similarly, the interactive renderer 15 receives the user orientation / position V L The regularly updated content of , and for each time-frequency slice, the modified transformation parameters W are determined and applied L .

[0124] In some embodiments, it is desirable to use different formats of transformation parameters W for different DOFs. p For example, listeners may be less sensitive to inaccuracies in the presented presentation when changing orientation / position on certain specific DOFs than on other DOFs. For rotational orientation in particular, it has been found that listeners are less sensitive to inaccuracies when changing pitch orientation (e.g., looking up or down) than when changing yaw orientation (e.g., looking from left to right). It has been found that most binaural localization cues (such as the interaural time and level differences that users use to localize sounds) remain constant with changes in pitch because their perceived position moves along the so-called cone of confusion.

[0125] The most salient cues for pitch changes are in the form of spectral changes, whereas the most salient cues for yaw changes are in the form of different interaural level differences and time delays. Therefore, it is envisaged that the transformation parameters W associated with different pitch orientations for a particular yaw orientation can be described using fewer parameters than for a particular yaw orientation. p .

[0126] For example, the transformation parameters W as described above may be determined for the set of principal yaw orientations: p , where the dominant yaw orientation has different yaw angles at predetermined pitch angles (e.g., zero pitch corresponding to a listener viewing horizontally). For at least one dominant yaw orientation, a simplified transformation parameter set W is determined. pitch is used to transform the main yaw orientation at a predetermined pitch to a companion pitch orientation (e.g., a 15 degree pitch corresponding to a listener looking up from the horizon). For each frame and frequency band, the simplified transformation parameter W pitch includes the gain of each channel in the presentation format. For example, for binaural or stereo presentation formats, the simplified transform parameter W pitchIncludes the gain g for the left channel for each time-frequency slice and band pitch,u,l and the gain g for the right channel pitch,u,r If a total of U different companion pitch orientations are used for each main yaw orientation, the gain in the simplified transformation parameters is applied as follows

[0127]

[0128] Wherein, the index u=±1, ..., ±U indicates the accompanying pitch orientation index of the main yaw orientation index p, and u=0 indicates the predetermined pitch angle of the main yaw orientation. The simplified transformation parameter W is formed pitch Gain g pitch,u,l and g pitch,u,r It may be obtained, for example, as the square root of the ratio of the signal energy level (eg, average energy level) of the rendered representation with pitch index u and dominant yaw index p to the representation with dominant yaw index p and predetermined pitch for each time-frequency tile.

[0129] exist Figure 6 , which schematically illustrates the use of format V p,u Multiple main yaw orientations V 1,0 、V 2,0 , where p is the yaw angle index and u is the pitch angle index. Each main yaw orientation V 1,0 、V 2,0 is associated with a predetermined pitch (e.g., a pitch of 0 degrees) and a corresponding set of transform parameters W1, W2 for transforming the primary presentation R1. Each set of transform parameters W1, W2 indicates an initial least squares solution transform matrix for each time-frequency tile The matrix elements of , or each set W p The enhanced transformation matrix M can be indicated p and decorrelation gain G d,p , which can be used to transform the main presentation R1 into a main yaw orientation V 1,0 、V 2,0 The yaw associated presentation.

[0130] For each main yaw orientation V 1,0 、V 2,0 , generate at least one simplified transformation parameter set W pitch , the simplified transformation parameter indicates the main yaw orientation V 1,0 、V 2,0 How should the transformation parameters W1 and W2 of the image be modified to obtain the transformation parameters that transform the transformation parameters W1 and W2 into the rendering with different pitches? pitch,2,1 and W pitch,2,-1 and the accompanying pitch orientation V 2,1 and V2,-1 The accompanying pitch orientation is related to the main yaw orientation V 2,0 In association, the main yaw orientation is associated with a transformation parameter W2.

[0131] By comparison Figure 6 and Figure 5 , it can be seen that for many pitch and yaw orientations, using the simplified transformation parameter W pitch Instead of parameter W p . Due to the transformation parameter set W p Each simplified transformation parameter set W is compared with at least four real or complex values pitch It only includes two real-valued gain values, namely g pitch,u,l and g pitch,u,r (or one per channel in a common presentation format), so much less data is needed to span an equal number of orientations, or more orientations can be represented with the same amount of data. Also, due to the simpler, more data efficient, simplified transform parameter W pitch It is only used to span pitch changes to which the user is not very sensitive, so the reduction in data does not substantially degrade the user's quality of experience.

[0132] In an exemplary embodiment, the first listener orientation V 1,0 and the second listener orientation V 2,0 is the dominant yaw listener orientation at the respective predetermined first and second pitch orientations (eg, 0 degrees pitch). The first listener orientation V 1,0 and the second listener orientation V 2,0 is associated with a corresponding (complete) transformation parameter set W1, W2. Optionally, the first listener orientation V 1,0 and the second listener orientation V 2,0 One is associated with the main rendering and default transformation parameters.

[0133] At least one of the first listener orientation and the second listener orientation is related to the third listener orientation V 2,1 In association, the pitch orientation of the third listener orientation is different from the first listener position V 1,0 or the second listener position V 2,0 , but the yaw orientation is the same as the first listener position or the second listener position. The third listener orientation V 2,1 and the simplified transformation parameter W pitch,2,1 In association, the simplified transformation parameters have a value for each time-frequency slice and channel in the presentation format, which value can be used to adjust the first listener orientation V 1,0 Or the second listener orientation V 2,0 The interactive renderer obtains the user's orientation V L , and based on user orientation VL and the first listener orientation V 1,0 , the second listener orientation V 2,0 and the third listener orientation V 2,1 The orientation deviation value is determined based on the orientation of at least one of the above, and the simplified transformation parameter W is applied based on the orientation deviation value. pitch,2,1 For example, to determine the user orientation V L Closest to the third listener orientation V 2,1 , where the simplified transformation parameter W associated with the listener orientation is pitch,2,1 is applied to the primary rendered presentation R1. It should be noted that in addition to the second listener orientation V 2,0 In addition to the associated full transformation parameters W2, simplified transformation parameters W pitch,2,1 In general, it should be noted that when using the simplified transformation parameters W associated with the main transformation parameters W1, W2 pitch,2,1 When , the modified transformation parameters W applied to the main presentation are determined L The step of determining the best matching set of primary transform parameters W1, W2 (by selecting the closest set or by interpolation) and for said best matching set of primary transform parameters W1, W2, determining the best matching set of simplified transform parameters W pitch,2,1 , W pitch,2,-1 The modified transformation parameter W L As a combination of these best matching sets, this means that the best matching main transformation parameters W1, W2 and the best matching simplified transformation parameters W pitch,2,1 , W pitch,2,-1 Applied to the primary presentation R1 to obtain the output presentation.

[0134] exist Figure 7 , the multi-rendering encoder 13 and the interactive renderer 15 are shown alongside a parameter encoder 18 and a parameter decoder 19. The parameter encoder 18 may be part of the multi-rendering encoder 13 or provided externally. Similarly, the parameter decoder 19 may be part of the interactive encoder 15 or provided externally. As illustrated above, it is envisaged that the multi-rendering encoder 15 is implemented in a wearable device (e.g., a pair of earbuds or a pair of headphones). Since the bandwidth for communicating with the wearable device may be limited (e.g., the communication may be wireless, such as Bluetooth), it is beneficial if the amount of data transmitted to the interactive renderer 15 is limited. As explained above, this may be achieved by using simplified transform parameters W for some listener positions. pitch As will now be described, the amount of data can be reduced by adjusting the transformation parameters W before and after transmission. p , W pitchEfficient encoding and decoding may alternatively or additionally be performed to reduce the amount of data transmitted to the interactive renderer 15 .

[0135] The parameter encoder 18 obtains the corresponding positions V1, ..., V P The associated transformation parameters W1, ..., W P (optionally including one or more simplified transformation parameter sets W pitch ). The transformation parameter sets (and any simplified transformation parameters) W1, ..., W P Combined into vector And for time-frequency slice based processing, a vector is generated for each time-frequency slice For example, if the three transformation parameter sets W p will be transmitted to the interactive renderer, and each transformation parameter set consists of four complex values ​​and one real value, then the vector 27 vector elements will be included. Any default transformation parameters W1 associated with the primary rendering R1 are optionally included, as these may already be stored in the interactive renderer 15 .

[0136] For each frequency band b, the parameter encoder 18 may use a plurality of predetermined basis vectors u b,n To approximate the vector Each frequency band has a total of N b basis vectors, which means that the index n=1, ..., N b . Basis vector N b The number of basis vectors may be the same across all frequency bands, or it is contemplated that different numbers of basis vectors may be used for different frequency bands. For example, for lower frequency bands, fewer basis vectors may be used.

[0137] Using basis vector u b,n , the parameter encoder 18 determines the approximation Vector The linear combination of the basis vectors of

[0138]

[0139] Among them, α n are real-valued coefficients. The basis vectors u are determined in advance using, for example, principal component analysis or similar data reduction methods. b,n , the method enables the use of a smaller set of coefficients α n to approximate the full vector. With the vector The number of elements in is compared to the number of basis vectors N b and real-valued coefficient α n Smaller.

[0140] Although it is certain that the vector can be accurately reconstructed The basis vector u b,n , but the parameter encoding performed by the parameter encoder 18 will generally not be lossless. On the other hand, in some embodiments, the residual vector is also determined by the parameter encoder 18, so that

[0141]

[0142] To this end, the residual vector can be selectively omitted when the bandwidth is limited and transmit when bandwidth permits or at least a portion thereof. Therefore, if the bandwidth for communicating with the interactive renderer 15 permits, the coefficient α may be n and the residual vector is transmitted to the interactive renderer 15, thereby allowing the vector Perfect reconstruction.

[0143] Coefficient α n and optionally the residual vector is included in the bit stream B, which is transmitted from the parameter encoder 18 to the parameter decoder 19. The parameter decoder 19 has stored the predetermined basis vectors u b,n , and use the above received coefficients and equation 22 to reconstruct the vector Optionally, if the residual vector is also available, then the parameter decoder 19 uses the above equation 23 to reconstruct the true transform vector Reconstructed vector Indicates transformation parameters The parameter processor 17 may use the transformation parameters to form modified reconstruction parameters W L Alternatively, if Can be rebuilt, then Indicates the original transformation parameters W1, ...W P .

[0144] In some embodiments, the indication and transformation parameters W1, ...W P The associated orientation / position V1, ... V P The data is transmitted together with the transformation parameters. For example, indicating the orientation / position V1, ... V P The data can be incorporated into the vector For example, the transformation parameters W1, ...W have been determined for it. P Rendering orientation / position V1, ...V P The number and relative orientation / position of may vary over time. It is also contemplated that positions V1, ...V P It may be predetermined and available, for example stored locally on the interactive renderer 15, as mentioned above.

[0145] Figure 8 A flowchart describing a method for processing audio according to some embodiments is depicted. Figure 1 The method comprises receiving at least one input signal at a second processing module at step S1. The first processing module may be a multi-rendering renderer 11 which receives one or more input audio signals from a database 10 with audio assets. The method then proceeds to step S2 which comprises generating a main rendered rendering R1 and generating at least one additional rendering R2, ..., R3 based on the at least one input audio signal. P , where P ≥ 2. A main presentation R1 and at least one additional presentation R2, ..., R P with different corresponding listener orientations V2, ..., V P Associated.

[0146] At step S3, for each additional presentation R2, ..., R P Determine the transformation parameter set W2, ..., W P Each transformation parameter set W2, ..., W P is configured to modify the main presentation R1 into an additional presentation W p . Further references Fig. 9 It is envisaged that step S3 involving determining transformation parameters may include: step S31, said step comprising determining transformation parameters W for additional presentation p , wherein the additional presentation is associated with a different yaw orientation compared to the main presentation; and step S32, the step comprising determining a simplified transformation parameter W for the second additional presentation pitch , the second additional presentation is associated with the same yaw orientation as the additional presentation but with a different pitch orientation. That is, the method comprises determining one or more sets of transformation parameters spanning a range of different dominant yaw orientations with a predetermined pitch, and subsequently determining one or more simplified sets of transformation parameters W for at least one dominant yaw orientation. pitch , describing how a presentation at a dominant yaw orientation with a predetermined pitch may be transformed into a presentation with the same yaw orientation but with a different pitch. A simplified set of transform parameters in the transform parameters indicates real-valued gains for each channel, while a (unsimplified) set of transform parameters indicates at least an N by N matrix, N being the number of channels in the presentation.

[0147] Return to Figure 8 The method then proceeds to step S4, which comprises transmitting the transformation parameters and the main rendering R1 to a second processing module. The second processing module may implement the interactive renderer 15 as described above. The interactive renderer 15 obtains an indication of the user orientation / position V LOr user orientation / position data at least indicative of the orientation / position of the user's head.

[0148] At step S5, the interactive renderer 15 determines the orientation of the user and the relationship between the primary presentation R1 and the transformation parameters W2, ..., W P The associated listener positions V1, ..., V P Based on the orientation / position deviation value, the interactive renderer 15 determines at step S6 to shift the main presentation R1 from the first listener orientation / position V1 to the user orientation / position V L The modified transformation parameter W L Determine the modified transformation parameters W L This may include selecting the closest user orientation / position V L Orientation / position V p The associated transformation parameter set W p , or at a position V corresponding to the user's orientation / position L At least two sets of transformation parameters W associated with nearby listener orientations / positions p , …, W P Interpolate between .

[0149] At step S7, the modified transformation parameter W L Applied to the primary presentation R1 to form the output presentation.

[0150] In order to facilitate efficient, low bit rate communication between the first processing module and the second processing module, if the transformation parameters W2, ..., W P This is beneficial if encoding is performed. Fig.10 is a diagram showing the Figure 8 Detailed embodiment of the flowchart of step S4. Transformation parameters W2, ..., W P Combined into vector At step S41, the vector describing the transformation parameters By predetermining the basis vector u b,n It is approximated by a linear combination of n This approximation of the indication can be called using the basis vector u b,n Pair Vector Encode. Basis vector u b,n The number is less than the vector The number of elements in , which means that the scalar α n Can be expressed as of An imperfect reconstruction.

[0151] At step S42a, the scalar α nis transmitted to the second processing module. Optionally, at step S42b, it is determined that the description is called of The residual vector of the difference between And transmit it to the second processing module.

[0152] At the second processing module, the scalar α n (and optionally the residual vector ) is used to use the same predetermined basis vector u at step S43 b,n To rebuild (or if Rebuild or The process of decoding the encoded transform parameters may be referred to as decoding the encoded transform parameters.

[0153] Unless otherwise specifically stated, it will be apparent from the following discussion that it should be understood that throughout the disclosed discussion, terms such as "processing," "computing," "calculating," "determining," "analyzing," etc. are utilized to refer to the actions and / or processes of computer hardware or computing systems or similar electronic computing devices to manipulate and / or transform data represented as physical (e.g., electronic) quantities into other data similarly represented as physical quantities.

[0154] It should be understood that in the above description of the exemplary embodiments of the present invention, various features of the present invention are sometimes combined together in a single embodiment, figure or description thereof to simplify the present disclosure and help understand one or more of the various creative aspects. However, the method of the present disclosure should not be interpreted as reflecting the intention that the claimed invention requires more features than the features explicitly stated in each claim. On the contrary, as reflected in the following claims, the creative aspects are less than all the features of the single aforementioned disclosed embodiment. Therefore, the claims following the specific embodiments are hereby expressly incorporated into the specific embodiments, wherein each claim is independently a separate embodiment of the present invention. In addition, although some embodiments described herein include some features included in other embodiments without including other features included in other embodiments, as will be understood by those skilled in the art, the combination of features of different embodiments is intended to be within the scope of the present invention and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0155] In addition, some embodiments herein are described as a combination of methods or method elements that can be implemented by a processor of a computer system or by other devices that perform functions. Therefore, a processor with an instruction for performing such a method or method element forms a device for performing a method or method element. It should be noted that when the method includes multiple elements (e.g., several steps), unless otherwise stated, any order of these elements is not implied. In addition, the elements of the device embodiments described herein are examples of devices that perform functions performed by elements in order to perform embodiments of the present invention. In the description provided herein, many specific details are set forth. However, it should be understood that embodiments of the present invention can be practiced without these specific details. In other examples, well-known methods, structures, and techniques are not shown in detail to avoid blurring the understanding of this specification.

[0156] Therefore, while particular embodiments of the present invention have been described, those skilled in the art will recognize that other and further modifications may be made thereto without departing from the spirit of the invention, and it is intended to claim all such changes and modifications that fall within the scope of the invention.

[0157] Various aspects of the present invention may be understood from the following enumerated example embodiments (EEE):

[0158] EEE 1.1. A method for processing audio, the method comprising: receiving an audio input at a first processing module, the audio input comprising an audio channel, an object, metadata, or a combination thereof; generating one or more rendered presentations and presentation transformation data of the audio input at the first processing module; receiving user interactivity data and the one or more rendered presentations and presentation transformation data generated by the first process at a second processing module; and generating an output presentation at the second processing module in response to the received user interactivity data, rendered presentation, and presentation transformation data.

[0159] EEE 1.2. The method according to EEE 1.1, wherein the output presentation is configured for headphone playback. EEE 1.2.

[0160] EEE 1.3. A method according to EEE 1.1 or 1.2, wherein the user interactivity data indicates a head orientation or position of the user. EEE 1.3.

[0161] EEE 1.4. The method according to any one of the preceding EEEs, wherein the two processing modules are implemented on different devices with different processing capabilities and / or processing delays.

[0162] EEE 1.5. The method according to any of the preceding EEEs, wherein the rendering transformation data represents a gain matrix or an input-output matrix having real-valued or complex-valued coefficients. ...

[0163] EEE 1.6. The method according to any of the preceding EEEs, wherein processing is applied according to time and frequency.

[0164] EEE 1.7. The method according to EEE 1.1, wherein the first process is divided into two sub-processes, the first sub-process is a renderer that renders multiple renderings, and the second sub-process is used to generate rendering transformation data.

[0165] EEE 1.8. The method according to any of the preceding EEEs, wherein the second process comprises a decorrelator stage, the decorrelator stage output being mixed into the output presentation with a gain dependent on the presentation transform data. ...

[0166] EEE 1.9. A method according to any one of the preceding EEEs, wherein the user interactivity data comprises (represents) a yaw angle and a pitch angle of the user's head, and the rendering transformation data contains data elements for two or more yaw angles and / or pitch angles.

[0167] EEE 1.10. A method according to EEE 1.9, wherein the data elements of the yaw angle and the pitch angle are represented as a yaw contribution and a pitch contribution, respectively. ...

[0168] EEE 1.11. A method according to any one of the preceding EEEs, wherein the rendering transformation data is represented by a predetermined set of basis functions and a set of basis function weights. EEE 1.12.

Claims

1. A method for processing audio, the method comprising: receiving at least one input audio signal at a first processing module; generating at said first processing module a primary rendered presentation and an additional rendered presentation, each rendered presentation being associated with a first listener orientation and / or position and a second listener orientation and / or position, respectively; determining, at the first processing module, transformation parameters for transforming the primary rendered presentation into the additional rendered presentation; receiving, at a second processing module, the transformation parameters and the primary rendered presentation generated by the first processing module; receiving, at the second processing module, user orientation and / or position data indicative of the orientation and / or position of the user; determining, at the second processing module, a deviation value based on the orientation and / or position of the user and at least one of the first listener orientation and / or position and the second listener orientation and / or position; determining, at the second processing module, modified transformation parameters based on the transformation parameters and the bias value; as well as The modified transformation parameters are applied to the primary rendered presentation at the second processing module to generate an output presentation associated with the orientation and / or position of the user.

2. A method according to any one of the preceding claims, wherein: Determining the transformation parameters at the first processing module includes: A transformation matrix having N times N elements is determined at the first processing module N is the number of audio channels in the primary rendered representation and the additional rendered representation, the transformation matrix Indicating a linear combination of the audio channels in the primary rendered presentation similar to the additional rendered presentation.

3. The method according to claim 2, wherein: Determine the transformation matrix include: Minimize the additional rendered representation with the transformation matrix The error between the primary rendered representations of the transform.

4. The method according to claim 2 or claim 3, wherein: The transformation matrix The elements of are real or complex values.

5. The method according to any one of claims 2 to 4, wherein: Determining the transformation parameters at the first processing module further comprises: Determine the enhanced modified transformation matrix M P , the enhanced modified transformation matrix M P is equal to the transformation matrix modified by the diagonal gain matrix G The diagonal gain matrix G is based on the transformation matrix A difference between the covariance of the primary rendered presentation and the covariance of the additional rendered presentation is modified.

6. A method according to any one of the preceding claims, wherein: The modified transform parameters define N decorrelation gains, N being the number of audio channels in the primary rendered representation and the additional rendered representation, the method further comprising: processing the primary rendered representation with a decorrelator at the second processing module to obtain a decorrelated primary rendered representation; and applying the modified transform parameters comprises: The decorrelation gain is applied to each channel of the decorrelated primary rendered representation.

7. The method according to claim 6, wherein: The decorrelated primary rendered representation is a combination of all channels of the primary rendered representation processed with a decorrelation processor.

8. A method according to claim 6 or 7 when appended to claim 9, wherein: The decorrelation gain is based on the transformation matrix The covariance of the primary rendered presentation and the covariance of the additional rendered presentation are modified.

9. A method according to any one of the preceding claims, wherein: The first listener orientation and / or position differs from the second listener orientation and / or position in at least one of a pitch orientation, a yaw orientation, and a roll orientation.

10. The method according to any one of claims 1 to 8, wherein: The first listener orientation and / or position and the second listener orientation and / or position are different yaw orientations in respective first and second pitch orientations, the method further comprising: obtaining, at the second processing module, simplified transformation parameters associated with a third pitch orientation, the simplified transformation parameters being configured to transform the primary rendered presentation or the additional rendered presentation into a pitch rendered presentation having the third pitch orientation; and The simplified transformation parameters are applied at the second processing module to the primary rendered presentation based on an orientation bias value to generate the output presentation.

11. The method according to claim 10, wherein: The simplified transform parameters comprise real-valued gains for each audio channel of the output presentation.

12. The method according to claim 10 or claim 11, wherein: Obtaining simplified transform parameters comprises obtaining a separate set of simplified transform parameters for each of a plurality of frequency bands, and wherein applying the simplified transform parameters comprises: The simplified transform parameters for each frequency band are applied to the corresponding frequency band of the primary rendered representation.

13. The method according to any one of claims 10 to 12, wherein: Applying the simplified transformation parameters to the primary rendered representation based on the orientation bias value comprises: determining, at the second processing module, modified simplified transformation parameters based on the simplified transformation parameters and the orientation bias value; and The modified simplified transform parameters are applied to the primary rendered presentation at the second processing module to generate the output presentation.

14. The method according to claim 13, wherein: The simplified transformation parameters include: primary simplified transformation parameters for transforming the primary rendered presentation at a first yaw orientation and a first pitch orientation to a primary pitch presentation at the first yaw orientation and a third pitch orientation, and additional simplified transformation parameters for transforming the additional rendered presentation at a second yaw orientation and a second pitch orientation to an additional pitch presentation at the second yaw orientation and a third pitch orientation, the method further comprising: The modified simplified transform parameters are determined at the second processing module based on the primary simplified transform parameters, the additional simplified transform, and the orientation bias value.

15. The method according to claim 13 or claim 14, wherein: Determining the modified simplified transformation parameters includes: Interpolation is performed between the primary simplified transform parameters and the additional simplified transform parameters based on the orientation deviation value.

16. The method according to any one of claims 13 to 15, wherein: The first pitch orientation and the second pitch orientation are associated with default simplified transformation parameters, and determining modified simplified transformation parameters comprises: Interpolation is performed between the default simplified transformation parameters and the simplified transformation parameters based on the orientation deviation value.

17. The method according to any one of the preceding claims, further comprising: encoding the transform parameters at the first processing module as coefficients associated with a predetermined set of basis vectors; as well as The predetermined set of basis vectors is used at the second processing module to decode the encoded transform parameters.

18. The method according to claim 17, wherein: The basis vectors are determined via principal component analysis.

19. The method according to claim 17 or claim 18, further comprising: determining, at the first processing module, a residual vector indicative of a difference between the encoded transform parameters and the transform parameters; truncating the residual vector at the first processing module; and The encoded transform parameters are decoded at the second processing module using the predetermined set of basis vectors and the truncated residual vector.

20. A method according to any one of the preceding claims, wherein: The output presentation is configured for headphone playback.

21. A method according to any one of the preceding claims, wherein: The transform parameters include different transform parameters for each frequency band of a plurality of frequency bands.

22. The method according to any one of the preceding claims, further comprising: generating at said first processing module a second additional rendered presentation, said second additional rendered presentation being associated with a third listener orientation and / or position, determining, at the first processing module, second transformation parameters for transforming the primary rendered presentation into the second additional rendered presentation; receiving, at a second processing module, the second transformation parameters generated by the first processing module; as well as Modified transformation parameters are determined at the second processing module based on the transformation parameters, the second transformation parameters, and the bias value.

23. The method according to claim 22, wherein: Determining the modified transformation parameters comprises: Interpolation is performed between the transform parameters and the second transform parameters based on the deviation value.

24. A method according to any one of the preceding claims, wherein: The primary rendered representation is associated with default primary transformation parameters, and wherein determining the modified transformation parameters comprises: Interpolation is performed between the transformation parameters and the default primary transformation parameters based on the deviation value.

25. A method according to any one of the preceding claims, wherein: The first processing module and the second processing module are implemented on different devices having different processing capabilities and / or processing latencies.

26. The method according to claim 25, wherein: The second processing module is a wearable device, such as headphones, earbuds, wireless earbuds, true wireless earbuds, smart glasses, or VR / AR / XR head-mounted devices.

27. The method according to any one of the preceding claims, further comprising: receiving, at an additional processing module, the transformation parameters and the primary rendered presentation generated by the first processing module; receiving, at the additional processing module, user orientation and / or position data indicative of an orientation and / or position of a second user; determining, at the additional processing module, a second deviation value between the orientation and / or position of the second user and the orientation and / or position of the first listener and the orientation and / or position of the second listener; determining, at the additional processing module, second modified transformation parameters based on the transformation parameters and the second bias value; as well as The second modified transformation parameters are applied to the primary rendered presentation at the additional processing module to generate a second output presentation associated with the orientation and / or position of the second user.

28. A method according to any one of the preceding claims, wherein: Determining the deviation value includes weighting different angular components of respective user and listener orientations and / or positions in different ways based on expected perceptual impact on the rendered presentation.

29. A method according to any one of the preceding claims, wherein: Determining the deviation value includes weighting different linear components of respective user and listener orientations and / or positions in different ways based on expected perceptual impact on the rendered presentation.

30. A method according to any one of the preceding claims, wherein: Determining the deviation value includes weighting linear and angular components of respective user and listener orientations and / or positions differently based on an expected perceptual impact on the rendered presentation.

31. A computer program product comprising instructions which, when said program is executed by a computer, cause said computer to perform the method according to any one of claims 1 to 30.

32. A computer-readable storage medium storing the computer program according to claim 31.

33. A system comprising a first processing module in communication with a second processing module, wherein: The first processing module and the second processing module are configured to perform the method according to any one of claims 1 to 30.

34. The system of claim 33, wherein: The first processing module and the second processing module are implemented on different devices, and the different devices are configured to communicate via wireless and / or wired connections.

Images