Modal interpolation based on impulse responses to generate spatialized audio signals

Abstract

Systems, methods, software, and devices are disclosed herein that transform a spatial input into a modal output that includes learned modal components of impulse responses. A neural network interpolates the modal components of impulse responses based on a desired sound source direction represented in the spatial input. The learned modal components are then used to determine coefficients of an infinite impulse response filter used to convert anechoic audio to spatialized audio. The spatialized audio provides a directional effect to a listener from the desired sound source direction.

Description

Technical Field

[0001] This disclosure relates to the field of audio processing, and more specifically, to spatial audio technology. Background Technology

[0002] Spatial audio refers to the audio effect where, when headphones, speakers, or other sound sources are brought close to a listener's ears, the sound appears to come from a specific direction. Users are increasingly encountering spatial audio in virtual reality and augmented reality environments, multimedia applications, and gaming experiences, where immersive experiences are prevalent and in demand.

[0003] Spatialized audio is created by configuring impulse response (IR) filters to modify the anechoic audio signal based on the head correlation transfer function (HRTF) and / or room impulse response (RIR). The resulting spatialized audio signal, output by the IR filter, drives the audio components that create the sound waves heard by the listener. The IR filter physically alters the frequency and phase characteristics of the anechoic audio signal according to the desired HRTF or RIR, so that when the sound waves reach the listener's ears, they produce the impression of coming from the direction of the desired sound source.

[0004] Immersive auditory experiences provided by spatialized audio require high-density HRTF samples, making them difficult to scale. Machine learning methods have been proposed to estimate HRTF in arbitrary directions. One such solution trains a neural network to estimate the amplitude response of the HRTF based on the sound source direction. The interpolated amplitude response is then converted into a finite-time impulse response (FIR) filter using the inverse discrete Fourier transform (DFT) with minimum phase. The FIR filter can then be applied to the anechoic audio signal to generate a spatialized audio signal associated with the desired sound source direction.

[0005] Unfortunately, learning how to interpolate the amplitude response of HRTF has proven difficult for neural networks, resulting in suboptimal spatial audio quality. Furthermore, the computational complexity of implementing FIR filters and their reliance on large amounts of memory limit their appeal and practicality. Summary of the Invention

[0006] This paper discloses a technique for improving spatialized audio using a neural network that converts a spatial input into a modal output, the modal output including learned modal components of an impulse response. The neural network interpolates the modal components of the impulse response based on the desired sound source direction represented in the spatial input. The learned modal components are then used to determine the coefficients of an infinite impulse response filter used to convert the anechoic audio into spatialized audio. The spatialized audio provides listeners with a directional effect from the desired sound source direction.

[0007] Neural networks can be implemented within the context of computing hardware and software systems such as personal computers, server computers, mobile phones, game consoles, and multimedia devices, which output spatialized audio via peripheral devices such as headphones, headsets, speakers, or others. Other suitable contexts include the peripheral devices themselves, such as headphones capable of executing neural networks. In fact, neural networks can be used to generate spatialized audio for a wide range of applications, such as virtual reality and / or augmented reality, games, and multimedia applications.

[0008] This overview is provided to introduce, in a simplified form, the selection of concepts that will be further described in the technical disclosure. It is understood that this overview is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0009] Many aspects of this disclosure can be better understood by referring to the following drawings. The components in the drawings are not necessarily drawn to scale, but are emphasized to clearly illustrate the principles of this disclosure. Furthermore, in the drawings, the same reference numerals consistently denote corresponding parts in multiple views. While multiple embodiments are described in conjunction with these drawings, this disclosure is not limited to the embodiments disclosed herein. Rather, it is intended to cover all alternatives, modifications, and equivalents. Attached Figure Description

[0010] [ Figure 1 ]

[0011] Figure 1 A spatialized audio system in implementation is shown.

[0012] [ Figure 2 ]

[0013] Figure 2 An implementation of an audio processing method is shown.

[0014] [ Figure 3A ]

[0015] Figure 3A An example of operation in an implementation is shown.

[0016] [ Figure 3B ]

[0017] Figure 3B An example of operation in an implementation is shown.

[0018] [ Figure 4 ]

[0019] Figure 4 Another example of operation in one implementation is shown.

[0020] [ Figure 5 ]

[0021] Figure 5 Another example of operation in one implementation is shown.

[0022] [ Figure 6A ]

[0023] Figure 6A The training environment in one implementation is shown.

[0024] [ Figure 6B ]

[0025] Figure 6B The training environment in one implementation is shown.

[0026] [ Figure 7 ]

[0027] Figure 7 The sample dataset and training data in one implementation are shown.

[0028] [ Figure 8 ]

[0029] Figure 8 The training process in one implementation is shown.

[0030] [ Figure 9 ]

[0031] Figure 9 This illustrates a multi-object training process in one implementation.

[0032] [ Figure 10 ]

[0033] Figure 10 This illustrates object-specific training processing in one implementation.

[0034] [ Figure 11 ]

[0035] Figure 11 An implementation of an artificial neural network is shown.

[0036] [ Figure 12 ]

[0037] Figure 12 The output layer of a neural network implementation is shown.

[0038] [ Figure 13 ]

[0039] Figure 13 A data visualization in an implementation is shown.

[0040] [ Figure 14 ]

[0041] Figure 14Computational systems suitable for implementing the various operating environments, architectures, processes, scenarios, and sequences discussed below with reference to the other figures are shown. Detailed Implementation

[0042] The acoustics from the sound environment can be represented by a linear signal called impulse response (IR), which is the sound received at each location in response to an impulse sound (similar to a "click") generated from a specific source location within the environment. Two types of IR in audio processing are the room impulse response (RIR), which measures the IR between two points in the environment, and the head-related transfer function (HRTF), which measures the IR from a location in free space to a listener's ear, modeling the effects on the listener's body, head, and ear shape. RIR and HRTF, along with other applications, are beneficial for immersive audio in augmented reality / virtual reality but are difficult to collect in practice.

[0043] In general, in the context of signal processing, a system's impulse response is the system's output under the condition of being subjected to an impulse input. In the time domain, the system's impulse response represents the system's behavior over time. In the frequency domain, the impulse response is associated with the system's transfer function. Since neural networks are mathematical models learned from data used to describe the spatial-temporal activity patterns of physical quantities, it is not surprising that they have recently been used to provide a mapping from position coordinates to the IR. Existing methods focus on directly estimating the IR in the time domain as a finite impulse response (FIR) filter or estimating the amplitude spectrum of the IR after it has been transformed into an FIR filter.

[0044] An FIR filter is a digital filter that has a finite-duration response to an input signal. The term "finite impulse response" refers to the fact that the filter's output is determined by a finite number of past input samples, which are weighted and summed to produce the current output. However, FIR filters (whose coefficients can be computed from the output of a neural network) offer an expensive IR representation in terms of computational and memory requirements. In practice, filters designed to transform anechoic audio signals into spatialized audio signals in this approach will have hundreds or even thousands of non-zero coefficients and must maintain large buffers of past samples, requiring significant memory usage.

[0045] Therefore, the audio processing systems, software, methods, and apparatus disclosed herein utilize neural networks trained to spatially interpolate the modal components of the impulse response based on the sound source directions included in the spatial input to the neural network. The learned modal components represent the design parameters of an infinite impulse response (IIR) filter, which are then converted into the coefficients of the IIR filter. IIR filters are characterized by an impulse response that extends infinitely into the past. The name "infinite impulse response" reflects the fact that the filter's output is determined by a finite sum of past input samples and past output samples, which is equivalent to an infinite sum of past input samples. IIR filters have a recursive structure, meaning that the output at a given time depends not only on the current input but also on previous output samples. This recursive property often results in more compact filter implementations compared to FIR filters. For example, while FIR filters have a large number of non-zero coefficients, IIR filters require far fewer coefficients to approximate an IR, making IIR filters advantageous in terms of performance, memory usage, and computational load.

[0046] The resulting IIR filter configuration converts the anechoic audio signal into a spatialized audio signal, which provides the listener with an auditory directional effect of sound from the desired source direction when the sound is actually produced by a sound source close to the listener. This technical effect stems from an understanding of the modal properties of IR, including RIR and HRTF, and can be approximated based on modal analysis. Modal analysis is a technique used to identify and study the vibrational modes of a system. A mode is the natural vibrational pattern exhibited by a system when excited. Modal analysis can be used in acoustics to understand the frequency-dependent behavior of sound propagation. In the case of IR, modes are typically represented by peaks and troughs of the transfer function, typically called resonant peaks / troughs, which represent the frequencies at which geometry and material properties cause sound accumulation / cancellation. Therefore, the embodiments disclosed herein are based on the understanding that the impulse response of a room or other space (e.g., the human head in the case of HRTF) can be analyzed according to its modal components (i.e., resonant peaks / troughs).

[0047] For this purpose, the neural network can be trained using the spatial coordinates of a specified sound source and listener location as input to output parameters characterizing the resonant modes of an IR (e.g., RIR or HRTF). Examples of these parameters include the center frequency and bandwidth of different vibrational modes of the IR. Some implementations are based on identifying, testing, and demonstrating that the spatial-temporal parameters of the IR's mode decomposition are sufficient to design a stable IIR filter with fewer parameters than the corresponding FIR filter. As a result, the combination of the neural network (trained accordingly) and the IIR filter produces a computationally and memory-efficient system for transforming anechoic audio signals into spatialized audio signals. For example, some implementations are able to produce filters with only a few dozen coefficients instead of the hundreds or even thousands of coefficients required for FIR filters calculated using existing techniques.

[0048] Furthermore, such a neural network can be trained on a per-object basis based on an HRTF specifically measured for individual users, and / or on an HRTF that is generally representative of a group of people and therefore suitable for the intended purpose. In other words, not only does the technique disclosed herein generally improve the practicality of spatialized audio, but it also increases the granularity at which spatialized audio can be provided to listeners. Indeed, as will be understood from the discussion below, the ease with which object-specific neural networks can be trained makes this approach particularly practical and desirable.

[0049] The neural network considered in this paper can be trained based on spatial input to generate modal output. The spatial input includes the source directions of multiple sound samples, while the modal output includes the learned modal components of the impulse response. For example, HRTF measurements can be performed at the listener's location relative to multiple sound sources at different locations relative to the listener's location. Therefore, the source direction of each HRTF sample is different relative to the source direction of each other sample. The source direction can be represented by the azimuth and elevation angles of the measured sound reaching the listener's location. The spatial input can optionally include the distance from the source direction to the listener's location, the object identity associated with that listener, and other suitable inputs.

[0050] Training the neural network includes: providing the direction of a sound source as input to the neural network, and obtaining an output from the neural network, the output including learned modal components of an estimated impulse response (e.g., HRTF) for the direction of the sound source. Training also includes determining coefficients of an IIR filter based on the learned modal components, and then determining an estimated frequency domain amplitude response of the estimated impulse response based on the coefficients.

[0051] Training continues by comparing the estimated amplitude response with a known frequency domain amplitude response (e.g., the HRTF of a sampled individual or a representative HRTF of a specific individual or group) to the measured impulse response, and updating the weights in the neural network based on the comparison results. For example, the estimated and known amplitude responses are fed as inputs to a loss function that outputs a feedback signal to the neural network. The parameters of the neural network (e.g., weights and biases) are adjusted according to known techniques until training is complete.

[0052] During inference, the neural network receives spatial input and produces learned modal components as output. These learned modal components are converted into filter coefficients for configuring IIR filters. The IIR filters process the anechoic audio based on these coefficients to produce spatialized audio. When output to an audience as audible sound, spatialized audio creates the effect that the sound arrives from somewhere other than its actual source. Although the sound can be output through a headset, headphones, or speaker located at the audience's position, the effect is that the sound arrives from a desired source direction or virtual source direction that has an angle of arrival offset relative to the listener's position. For example, the audio may sound as if it originates from the user's left (or right), behind (or in front) the user, or from some other direction. This effect may also include a distance component, making the audio sound as if it originates near the user, far away from the user, etc.

[0053] In some implementations, spatialized audio is tailored to each listener based on their object (audience) identity during training, allowing object identity or object parameters to influence the neural network during inference. In one example, HRTF samples are collected for multiple objects, one for each object. Although the sound source orientation is encoded into the feature vector provided to the input layer of the neural network, object identity and / or object parameters can be provided via separate input channels. Object identity and / or parameters can be used to control which parameters of the neural network are updated for a given object. The neural network is able to learn from the training data to interpolate modal components based on object identity and / or parameters.

[0054] The result of multi-object training is, in a sense, multiple neural networks (or parameter sets) corresponding to each object. For example, a base network can be trained on the HRFT of a first user. Then, the base network for a second, subsequent user can be updated based on that user's HRFT, resulting in an updated version of the neural network without overwriting or replacing the parameters of the base network. During inference, the base neural network is loaded with respect to the first user to produce spatialized audio customized based on that user's HRFT. Similarly, a second version of the neural network is loaded with respect to the second user to produce spatialized audio customized based on that user's HRFT.

[0055] In some cases, a base network can be trained on the HRTF of a group of users. That is, instead of a single base user, there is a group with multiple users and multiple HRTFs corresponding to said multiple users. A network can be trained on the corresponding HRTF for each user in the base user group. Then, the base network can be updated for the next user based on the HRTF of the next user after the base user group (after it has already been trained on the base users), resulting in an updated version of the neural network, but without overwriting or replacing the parameters of the base network. At inference time, the base neural network is loaded with respect to any user in the base group to produce spatialized audio customized based on that user's HRTF. Similarly, an updated version of the neural network is loaded with respect to the next user to produce spatialized audio customized based on that user's HRTF.

[0056] It is understood that the techniques disclosed herein for transforming anechoic audio signals into spatialized audio signals are also applicable to transforming audio signals with some existing spatialization into audio signals with increased spatialization. In fact, the anechoic audio signals mentioned throughout the text may inherently include some spatialization characteristics. That is, since anechoic signals with absolutely no reflections or echoes are difficult (if not impossible) to achieve in practice, the term "anechoic" means that the audio signal (if not completely anechoic) is substantially less spatialized than the spatialized audio signal generated according to the disclosed implementation. Therefore, the term "anechoic audio signal" as used throughout the text refers to both purely anechoic audio signals and audio signals that are provably anechoic relative to the spatialized audio signal generated according to the disclosed implementation.

[0057] Please see the attached image. Figure 1 A spatialized audio system 100, hereinafter referred to as system 100, is illustrated in an implementation. System 100, which may be implemented in computer hardware, software, and / or firmware, includes a neural network 101, a conversion module 103, and an IIR filter 105. The neural network 101 is operatively connected to the conversion module 103, which in turn is operatively connected to the IIR filter 105. This connection may include providing a specific value, with the output as input, to a next element. For example, the neural network outputs learned modal components to the input of the conversion module 103, which outputs filter coefficients for configuring the IIR filter 105.

[0058] Neural network 101 represents an artificial neural network or other such machine learning algorithm capable of processing spatial input and producing modal output. Conversion module 103 represents an arbitrary functional block capable of converting modal components into IR filter coefficient values. IIR filter 105 represents a cascaded audio filter capable of processing anechoic audio to produce spatialized audio based on coefficients determined by conversion module 103. For example, IIR filter 105 may comprise a cascade of multiple IIR filter sections, each corresponding to a significant perceptual characteristic (such as peak, valley, or roll-off) in the spectrum modeled by the modal components output by neural network 101. Neural network 101, conversion module 103, and IIR filter 105 may each be implemented by software or firmware executed by circuitry on a single computing device or distributed across one or more processing devices. (Alternatively or additionally, some or all of the functionality provided by any of neural network 101, conversion module 103, and IIR filter 105 may be implemented entirely via application-specific integrated circuits or other such dedicated processing devices.)

[0059] Figure 2 An example is an audio processing 200 that uses system 100 to generate spatialized audio. Audio processing 200 can be implemented in the context of software and / or firmware components of system 100 (e.g., neural network 101, conversion module 103, and IIR filter 105) via program instructions. When executed by one or more processing devices of one or more suitable computing devices, the program instructions instruct those computing devices to operate as follows, with reference to the following for clarity: Figure 2 The steps and the computing device is indicated in the singular.

[0060] In operation, the computing device provides spatial input to the neural network 101 (step 201). The spatial input includes the desired sound source direction relative to the audience's position. For example, the spatial input may indicate the direction of the sound source 111 in the virtual or augmented reality environment 110 relative to the audience's position 113. The relative position may be indicated by elevation and azimuth angles determined by the two positions. The relative position may be provided by an upstream application or component (such as a virtual / augmented reality application, multimedia application, game application, etc.) capable of dynamically determining the direction as the relative position changes in real time. In other cases, the direction may be a pre-determined and pre-programmed static value. The computing device provides the spatial input in the form of a feature vector in which the sound source direction is encoded. Other information (such as distance and / or object identity) may also be encoded into the feature vector or provided as input to the neural network. Alternatively, the neural network 101 may be trained to encode the spatial input into the feature vector, in which case the spatial input may be input into the network in an encoded format.

[0061] Next, the computing device executes neural network 101 to obtain the learned modal components based on the spatial input (step 203). As described above, neural network 101 can be an artificial neural network. In this implementation, neural network 101 includes an input layer, one or more hidden layers, and an output layer. Feature vectors representing the spatial input are fed into the input layer of the model. The feature vectors can be generated using randomized Fourier feature (RFF) mapping or other suitable mechanisms.

[0062] The input layer consists of n input nodes corresponding to the n-dimensional feature vector. Each input node processes a corresponding part of the feature vector and passes its output to the hidden layer of the neural network. Each hidden layer takes the output of the previous layer as input and provides the output to the next layer of the neural network. The output layer of the neural network takes the output of the last hidden layer as input and its own output represents the values ​​of the modal components of the impulse response.

[0063] Next, the computing device executes conversion module 103 to convert the learned modal components output by the neural network into filter coefficients that control the behavior of IIR filter 105 (step 205). As described above, IIR filter 105 is typically represented as a cascade of multiple IIR filter sections, where each IIR filter section corresponds to an important perceptual characteristic (such as peak, valley, or roll-off) in the spectrum modeled by the modal components output by the neural network. In some implementations, the modal components output by the neural network are inputs to offline derivation equations for digital filter coefficients of a simulated prototype section having a desired filter shape (peak or frame) and a desired filter order (i.e., N). For example, for a bi-second-order filter, N = 2. Equations for the digital filter coefficients are derived offline using the well-known bilinear transform. Quantities provided by the neural network (gain, center frequency, and bandwidth – along with the sampling frequency of the digital signal), or intermediate values ​​calculated for convenience, are input into the equations in real time by the computing device to generate coefficients. Each filter coefficient represents a value used to configure the corresponding section of IIR filter 105. The computing device configures the IIR filter 105 using coefficient values, for example, by setting the parameters of the filter to values ​​(step 207).

[0064] Once configured, the computing device processes the silencing audio signal 115 using an IIR 105 filter to generate a spatialized audio signal 117 (step 209). The computing device can then output the spatialized audio signal as an audible sound that gives the listener the impression that the sound is coming from the desired (virtual) sound source direction.

[0065] Figure 3A and Figure 3BAn operational example 300 of audio processing 200 is illustrated. Operational example 300 involves computing device 301 and audio device 303. Computing device 301 represents a personal computer, server computer, tablet computer, game console, mobile phone, wearable device, or any other suitable computing device capable of employing system 100 and performing audio processing 200. Audio device 303 represents any device capable of receiving spatialized audio signals and outputting audible sound, examples of which include headsets, headphones, and speakers.

[0066] In some cases, computing device 301 and audio device 303 can be provided within the context of a single device, such as virtual / augmented reality goggles or glasses. Alternatively, or additionally, in a client-server implementation, the functionality provided by computing device 301 can be distributed across multiple computing devices. In yet another alternative, or additionally, some or all of the functionality of computing device 301 can reside within audio device 303. That is, in some implementations, audio device 303 itself can utilize system 100 to perform audio processing 200.

[0067] exist Figure 3A In the portion of the operational example 300 shown, computing device 301 presents a virtual environment 310 on its screen. In this example, virtual environment 310 represents a game or virtual reality environment with a first-person perspective, in which user 305 navigates the environment via an avatar 313 on the screen. The avatar 313 is typically positioned at the center of the screen and can be controlled by the user to navigate the environment. Virtual environment 310 includes an object 311 located separately from and at a distance from the avatar 313. Sounds “emitted” by object 311 can be generated from a sound file used to present virtual environment 310. For example, sound file 306 can reside on computing device 301 so that it can be invoked and processed when triggered by object 311. The “sound” obtained by transmitting audio signals from computing device 301 to audio device 303 plays realistic analog sounds for user 305 to enjoy.

[0068] The desired outcome is that the realistic sounds produced by audio device 303 appear to user 305 as if they originate from the same direction as the object 311 in the virtual environment 310 relative to the avatar 313. Therefore, computing device 301 uses audio processing 200 to determine the sound source direction of the sound emitted by object 311 relative to avatar 313. The virtual direction of the sound is fed by computing device 301 into a neural network that outputs modal components of impulse signals. Computing device 301 converts these modal components into filter coefficients for configuring IIR filters.

[0069] Within computing device 301, a software component associated with object 311 generates a silencing audio signal from sound file 306. This silencing audio signal is sent to audio device 303 and played by the audio device without any modification. The resulting sound will lack the desired directional effect from the front and left of user 305. However, by using audio processing 200, computing device 301 is able to pass the silencing audio signal through an IIR filter to obtain a spatialized audio signal 307. Computing device 301 then sends the spatialized audio signal to audio device 303.

[0070] Audio device 303 receives and plays spatialized audio signals, for example, by driving a speaker assembly that generates sound waves with spatialized characteristics. When spatialized audio signal 307 is received from computing device 301, the spatialized audio signal 307 can be in digital or analog format. If it is in digital format, the audio device converts the signal into an analog signal and drives output elements based on the analog signal. When experienced by user 305, the spatialized sound provides a directional effect of sound from the front and left of the user.

[0071] exist Figure 3B In the example 300, it is further assumed that user 305 has navigated avatar 313 to the left of object 311. Therefore, the relative position of object 311 with respect to avatar 313 has changed, thus altering the sound source direction. Computational device 301 inputs the new sound source direction into a neural network to obtain new modal components. These new modal components are then converted into new filter coefficients. The new coefficients cause the IIR filter to generate a new spatialized signal 327, which provides a new directional effect 329 when the audio device 303 plays the audio. This new directional effect makes user 305 perceive the sound as arriving from the front and right, rather than... Figure 3A Arrive from the front and left, as shown in the middle.

[0072] exist Figure 3A and Figure 3B In this example, since operational example 300 is intended to illustrate the operational scenario, spatialized audio is immediately output from computing device 301 to audio device 303. However, it is understood that in a preprocessing scenario, system 100 and audio processing 200 are deployed, where spatialized audio is created in advance before it is downloaded and deployed to the user's computer, audio device, or both.

[0073] For example, in some implementations, system 100 and audio processing 200 can be used in a media production environment to produce spatialized audio for movies, television programs, video games, animated videos, video clips, and purely audio-based media without video (e.g., audiobooks, music, etc.). Figure 4 An example of this is shown.

[0074] exist Figure 4 In this example, operational example 400 involves a production server 401, a computing device 402, and an audio device 403. The production server 401 represents any computer capable of employing system 100 and performing audio processing 200. The computing device 402 represents any device capable of receiving spatialized audio data and outputting a spatialized audio signal represented by said data. The audio device 403 represents any device capable of receiving spatialized audio signals and outputting audible sound, examples of which include headsets, headphones, and speakers. In some cases, the computing device 402 and the audio device 403 may be integrated into a single device such as a smart TV with integrated speakers, a mobile phone, and its speakers.

[0075] During operation, production server 401 receives muted audio 406. Muted audio can be received as part of a multimedia file or file group (such as a movie, video clip, music recording, or other such audio source). In some cases, audio can be received separately from video. In other cases, audio may not include any video, such as for songs, audiobooks, etc.

[0076] Production server 401 also receives or acquires spatial input data. Spatial input data, for example, represents the assumed listener location at various points in time, thereby allowing production server 401 to generate spatialized audio data that is temporally synchronized with the assumed listener location. Alternatively or additionally, spatial input data may include the assumed or annotated location of sound source objects, such as elements on screen (e.g., airplanes in the sky in a movie) or musical instruments (e.g., orchestral instruments positioned differently relative to each other and relative to the assumed listener center location).

[0077] Therefore, production server 401 can determine the relative position of the sound source to the virtual audience at each point in time. For each point in time or for each duration of an acceptable length, production server 401 determines the modal components of the pulse signal based on the direction of the sound source at that time (or for that time period). Production server 401 then converts the modal components into filter coefficients, which are used to configure an IIR filter. The IIR filter filters the anechoic audio based on the coefficients, thereby producing spatialized audio data in an audio file format.

[0078] In operational example 400, for illustrative purposes, it is assumed that the virtual audience is located to the left of the sound source. For example, the sound source could be a plane or other such object that is typically located on the right side of the screen in a video. Therefore, the sound associated with the airplane is associated with a sound source direction to the right relative to the center position of the virtual audience.

[0079] Production server 401 provides rightward input as spatial input to the neural network to obtain learned modal components representing the design parameters of the IIR filter. Production server 401 converts the modal components into coefficients of the IIR filter. The IIR filter then processes the anechoic audio signal encoded in audio file 406 to produce a spatialized audio signal. The spatialized audio signal can then be encoded into audio file 407.

[0080] Production server 401 provides spatialized audio data from sound file 407 to computing device 402 for output by audio device 403. It is conceivable that production server 401 may provide spatialized audio data to computing device 402 indirectly, such as via one or more networks, through online distribution channels, or in any other way. Alternatively, or additionally, production server 401 may provide spatialized audio data directly to computing device 402. Similarly, sound file 407 may be provided by production server 401 as a standalone file, such as in the context of a movie or video clip, or as one of multiple files downloaded, either within a variety of transmission formats.

[0081] In any case, computing device 402 receives and stores sound file 407 for retrieval and playback at an appropriate time. For example, computing device 402 can retrieve and play sound file 407 at a specific moment in a video game, in the background of a movie or song, while playing a specific video clip, or at any time involving sound file 407.

[0082] In this example, it is assumed that sound file 407 represents the sound of object 411, which is an output of a familiar sound (e.g., the sound of an airplane flying through the sky) displayed in media environment 410. When object 411 is displayed on the screen of computing device 402 or at points in time surrounding it, the computing device 402 processes sound file 407 to generate spatialized audio signal 417. The computing device 410 then sends spatialized audio signal 417 to audio device 403 associated with user 405.

[0083] Audio device 403 receives and processes spatialized audio signal 417 to generate an analog signal that drives the device's speaker components to produce audible sound for user 405. Due to the spatialized nature of the audio signal, the audible sound provides a directional effect 409 of sound originating from the user's right. A similar effect can be provided with orchestral music, for example, by presenting the user with some instruments playing on their right and others on their left, as described above. Figure 4 These effects can be achieved in all the preprocessing environments discussed.

[0084] exist Figure 3A and Figure 3B as well as Figure 4 In this context, besides the audio devices that output the final sound, the computer is responsible for converting the anechoic signal into a spatialized signal. Figure 5 In this example, an operational example 500 is provided, in which system 100 and audio processing 200 are locally deployed with respect to an audio output device.

[0085] Operational example 500 involves computing device 501 and audio device 503. Computing device 501 represents a mobile phone, tablet computer, game console, personal computer, server computer, or any other suitable computing device capable of outputting anechoic audio signals and spatial data. Audio device 503 represents a headset (also known as an earpiece), headset, or speaker capable of processing anechoic audio signals and spatial inputs to generate and output spatialized audio signals.

[0086] In operation, computing device 501 presents a game environment 510 on its screen, but this can be any type of environment, such as a virtual reality environment or an augmented reality environment (or none at all in an audio-only scenario). The game environment 510 includes a first-person perspective in which the user 505 navigates the environment via a user control object 513 on the screen. The user control object 513 is typically located in the center of the screen and can be controlled by the user to navigate the environment. In this example, the game environment 510 is a racing game and the user control object 513 represents a racing car.

[0087] The game environment 510 includes objects 511 and 512, both of which are located separately from the user-controlled object 513. Sounds “emitted” by objects 511 or 512 can be generated by sound files used to render the game environment 510. For example, the sound files can reside on the computing device 501 so that they can be invoked and processed when triggered by object 511. The resulting anechoic audio signal 504 is transmitted via audio signals from the computing device 501 to an audio device 503, which plays realistic analog sounds for consumption by the user 505.

[0088] The audio device 503 is expected to produce sound as if it is arriving from the same direction as the object 511 (or 512) in the game environment 510 relative to the user-controlled object 513. Therefore, the computing device 501 also provides spatial data 506 to the audio device 503, allowing the audio device 503 to convert the anechoic audio signal 504 into a spatialized audio signal 507.

[0089] Audio device 503 uses system 100 and audio processing 200 to determine the sound source direction of a sound emitted by object 511 (or object 512) relative to user-controlled object 513. The virtual direction of the sound source provided to audio device 503 by computing device 501 is fed into a neural network executed on audio device 503, which outputs learned modal components of impulse signals. Audio device 503 converts the modal components into filter coefficients for configuring IIR filters. This silenced signal is then passed through the IIR filters to generate a spatialized audio signal. Audio device 503 further processes the spatialized audio signal to generate analog signals for elements that drive the device to create sound waves. The resulting sound waves provide a directional effect 509 of one race car overtaking another in the background of game environment 510.

[0090] Understandably, as the position of the user-controlled object 513 continues to change relative to objects 511 and 512, the computing device 501 can continuously update the audio device 503 with the new spatial data. In this scenario, the audio device 503 inputs the new sound source direction into the neural network to obtain new modal components. These new modal components are then converted into new filter coefficients. The new coefficients drive the IIR filter to generate a new spatialized signal that provides new directional effects.

[0091] In the above operational example, it is assumed that the neural network used to generate the learned modal components is trained accordingly. Figure 6A The training environment 600A used to train this model is shown.

[0092] The training environment 600A, which can be implemented in computer hardware, software, and / or firmware, includes a neural network 601, a transformation module 603, a response module 605, a loss function 607, an encoding module 611, and a response module 613. The neural network 601 is operatively connected to the encoding module 611, the transformation module 603, and the loss function 607. The training environment 600A also includes response modules 605 and 613.

[0093] The neural network 601 and other components of the training environment 600A can each be implemented using software or firmware executed by circuitry on a single computing device or distributed across one or more processing devices. Alternatively or additionally, some or all of the functionality provided by any of the components can be implemented entirely via application-specific integrated circuits or other such special-purpose processing devices.

[0094] Generally speaking, training environments like 600A and such Figure 1The runtime environment of system 100 is implemented separately because the model is trained in such an environment and then deployed to the runtime environment. However, it is understood that some or all of the training environment 600A may also be deployed in the runtime environment along with the trained or untrained model to provide further training of the model in the context of a local execution environment (e.g., on a personal computer, mobile phone, etc.).

[0095] Encoding module 611 provides feature vectors as input to neural network 601, which produces learned modal components as output. The feature vectors can be generated by encoding module 611 using a randomized Fourier feature (RFF) map or other suitable mechanisms. For example, in some implementations, encoding module 611 can generate feature vectors via a machine learning algorithm (e.g., via neural network 601 or a separate network). Neural network 601 provides the learned modal components as input to transformation module 603, which produces coefficient values ​​as output. Response module 605 takes the coefficient values ​​as input and produces an estimated frequency domain amplitude response as output, which is subsequently referred to as the estimated amplitude response.

[0096] Loss function 607 accepts the estimated amplitude response as input, and also accepts the sampled frequency domain amplitude response from response module 613, which is referred to hereafter as the sampled amplitude response. Response module 613 generates the sampled amplitude response using impulse response values ​​from the training dataset, while encoding function 611 encodes the feature vector using spatial data from the training set (and optionally other data).

[0097] Neural network 601 represents an artificial neural network or other such machine learning algorithm capable of processing spatial input and producing modal output. Encoding module 611 represents any functional block capable of encoding spatial data in a feature vector. Transformation module 603 represents any functional block capable of converting modal components into IR filter coefficient values. Response module 605 represents a functional block capable of generating an estimated frequency domain amplitude response of the filter based on the filter coefficients. Response module 613 represents a functional block capable of generating the amplitude response in the frequency domain of a sample impulse response. Loss function 607 represents any functional block capable of comparing the output of response module 605 with the output of response module 613 to generate feedback for adjusting the parameters of neural network 601.

[0098] Figure 6BTraining environment 600B is illustrated; it is a variant of training environment 600A. Except for the addition of object-specific parameters 612, training environment 600B is identical to training environment 600A and includes all the same elements. The object-specific parameters 612 represent features associated with a specific object that allows neural network 601 to train on multiple objects. In addition to the feature vectors fed into the input layer of the network, object-specific parameters 612 are provided as additional input to neural network 601. Object-specific parameters 612 are used to control how the biases and parameters of neural network 610 are configured and / or updated, such that the network learns to interpolate modal components based on each object, as discussed below in relation to multi-object training. Figure 9 and Figure 10 This point was discussed in more detail.

[0099] Figure 7 The components of training environment 600A and / or training environment 600B are shown, along with a dataset 700 used to train the model. Dataset 700 includes HRTF samples 710 and training data 720. HRTF samples 710 include samples obtained in the background of sampling environment 711. Each individual sample is created by placing sound sources at different locations (or by moving a single sound source to different locations) and generating audio pulse signals (sound waves) captured at discrete times by a microphone or other such sensor at listener position 715. Here, the different sound source positions are represented by positions 712, 713, and 714.

[0100] Using position 714 as an example, each sample is defined based on its coordinates in three dimensions (i, j, and k) of the sampling environment 711. The sound source direction of each pulse is calculated by creating the specific sound source position for each pulse based on the ijk coordinates relative to the listener position 715. The distance between the sound source position and the listener position can also be calculated. The sound source direction can be represented by the azimuth and elevation angles formed by the two positions (sound source position and listener position) recorded for each sample.

[0101] Training data 720 is created from samples acquired from sampling environment 711 and includes source direction information 721, HRTF information 723, distance information 725 (optional), and object identity or SID data 727. In other words, training data 720 for each individual sample includes: source direction; HRTF; distance (optional); and object identity. When executed by training processing 800, the elements of training environment 600A use training data 720 to train neural network 601.

[0102] Figure 8The training process 800 shown can be implemented using program instructions within the context of software and / or firmware components of the training environment 600A. When executed by one or more processing devices of one or more suitable computing devices, the program instructions direct said one or more computing devices to operate as follows, incidentally with reference to Figure 8 The steps are as follows. While training process 800 generally represents how a neural network 601 is trained, it is highly simplified and only provides a snapshot of the training process for a single training cycle and a single feature vector.

[0103] In operation, the computing device performing encoding function 611 extracts spatial data from the impulse response samples and encodes the spatial data in a feature vector (step 801). The feature vector can be generated based on, for example, a randomized Fourier feature (RFF) map or other suitable vectorization techniques. Optionally, other information such as distance can also be encoded into the vector. The computing device continues to input the feature vector into neural network 601 (step 803). Neural network 601 includes an input layer, one or more hidden layers, and an output layer.

[0104] Next, the computing device executes neural network 601 to obtain the learned modal components based on the spatial input (step 805). The input layer comprises n input nodes, corresponding to the n-dimensionality of the feature vector. The input nodes process the corresponding parts of the feature vector and pass their outputs to the hidden layers of the neural network. Each hidden layer takes the output of the previous layer as input and provides outputs to subsequent layers of the neural network. The output layer of the neural network takes the output of the last hidden layer as input, and its own output represents the value of the modal component of the impulse response.

[0105] The computing device then executes conversion module 603 to convert the learned modal components of the neural network output into filter coefficients (step 807). Each of the filter coefficients represents a value used to configure the corresponding part of the IIR filter. The coefficients are provided to response module 605, which determines the estimated amplitude response of the IIR filter based on the coefficients (step 809). This can be achieved, for example, by inputting the coefficient values ​​and the complex exponent at each frequency of interest into the filter's transfer function and taking the magnitude of the resulting complex number as the estimated amplitude response. The computing device provides the estimated amplitude response as input to loss function 607.

[0106] In some cases, as an alternative to directly converting the filter coefficients into an estimated amplitude response, the IIR filter can be configured based on the coefficients, providing an impulse response (HRTF) associated with the current training loop and the input. Using the impulse response as input, the IIR filter will produce an output signal that can be analyzed to determine the estimated amplitude response.

[0107] Regardless of how the estimated magnitude response is determined, loss function 607 compares the estimated magnitude response with the sampled magnitude response generated by response module 613 for HRTF samples associated with the feature vector (step 811). The result of the comparison provides feedback on the training of neural network 601. That is, the computing device updates the parameters and / or biases of neural network 601 based on the result of the comparison (step 813). Updating the parameters and / or biases involves changing some or all of the parameter and bias values, and further updates are avoided if the result of the comparison indicates that training is complete.

[0108] Figure 9 A multi-object training process 900 is illustrated in an implementation, in which a neural network can be trained to learn modal components based on each object. The multi-object training process 900 can be implemented as program instructions within the context of software and / or firmware elements of a training environment 600B. When executed by one or more processing devices of one or more suitable computing devices, the program instructions instruct said one or more computing devices to operate as follows, incidentally with reference to... Figure 9 The steps are as follows. It is understandable that while the multi-object training process 900 generally represents how the neural network 601 is trained, it is highly simplified and only provides a snapshot of the multi-object training process.

[0109] In operation, the computing device trains neural network 601 (step 901) with respect to the base user or base user group. Such training can be achieved by utilizing information about... Figure 8 The training process described is implemented using 800. The base training process can be object-agnostic because no object-specific parameters are used when updating the network's parameters and biases. However, in addition to the spatial data encoded in the vectors, the feature vectors will be encoded with the object ID associated with each base user.

[0110] Next, the computing device determines whether there are samples from other users after the base group (step 903). That is, once the model has been trained on the training data associated with the base users, the computing device determines whether to continue training the model for other users. If not, training is complete, and, where applicable, basic instances of the neural network can be deployed on the first user and / or other users in the base user group. However, if other non-base users still exist, the computing device continues to determine the object-specific parameters for the next user (step 905).

[0111] In some cases, object-specific parameters are represented by one-hot vectors uniquely associated with a particular object. Object-specific parameters allow the computing device to train only those next user-specific parameters or biases of the network, and / or train based on a per-object configuration (907). The updated version of the neural network is then saved in association with the current object for later use in association with that object (step 909). Processing returns to step 903 until no other objects remain. The training performed in step 907 is described below. Figure 10 The described object-specific processing.

[0112] In some implementations, object-specific parameters include an N-dimensional learned embedding vector for each object. In this case, updating the network parameters involves applying a Feature-by-Feature Linear Modulation (FiLM) layer to each hidden layer in the neural network. In the FiLM case, when adding a new object, only an additional N-dimensional learned embedding vector for that object needs to be added to the network, while the remaining parameters remain frozen to their base values. Alternatively, object-specific parameters can be bias vectors of a subset of the network's hidden layers. In this case, updating the neural network parameters for a given object involves updating only the subset of bias terms corresponding to that specific object. This approach is often referred to as BitFit.

[0113] In another alternative, the weight matrices of a subset of the network's hidden layers are object-specific parameters. However, instead of having the full weight matrices as object-specific parameters, a low-rank adaptive (LoRA) approach is used to fine-tune the network. Here, each weight matrix can be represented as a product of two low-rank vectors, which advantageously reduces the number of object-specific parameters that need to be stored. Then, to update the neural network parameters for a specific object, only the weight matrices computed as low-rank products of the vectors corresponding to that object are updated.

[0114] Understandably, for each subsequent object after the base object, a relatively small number of HRTF samples are needed to update the network for the next object. That is, some parameters of the neural network are frozen for subsequent objects, while only object-specific parameters are updated. Figure 10 A brief illustration of an object-specific training process 1000 is provided, which is applicable when training a neural network 601 with respect to a base object followed by training a neural network 601 with respect to a subsequent object.

[0115] The object-specific training process 1000 can be implemented using program instructions within the context of software and / or firmware components of the training environment 600B. When executed by one or more suitable computing devices or processing devices, the program instructions instruct the one or more computing devices to operate as follows, incidentally referring to... Figure 10The steps are as follows. It is understandable that while object-specific training process 1000 generally represents how the neural network 601 is trained, it is highly simplified and only provides a snapshot of the object-specific training process.

[0116] In operation, after the neural network 601 has been trained on training data associated with a base object or a group of base objects, the computing device freezes the parameters of the neural network 601 in its current state (step 1001). Next, the computing device inputs an object-specific feature vector into the input layer of the neural network 601 (step 1003). The object-specific feature vector encodes at least the spatial input and the object ID associated with a given audience.

[0117] The computing device executes a neural network 601 on an object-specific feature vector to obtain learned modal components (step 1005). Then, the computing device executes a transformation module 603 to convert the learned modal components output by the neural network into filter coefficients (step 1007). Each filter coefficient represents a value for a corresponding part used to configure the IIR filter. The coefficients are provided to a response module 605, which determines an estimated magnitude response of the IIR filter based on the coefficients (step 1009). The computing device provides the estimated magnitude response as input to a loss function 607.

[0118] Loss function 607 compares the estimated magnitude response with the sampled magnitude response generated by response module 613 for HRTF samples associated with the feature vector (step 1011). The result of the comparison provides feedback on the training of neural network 601. That is, the computing device updates the user-specific parameters and / or biases of neural network 601 based solely on the result of the comparison (step 1013).

[0119] Figure 11 Artificial neural network 1100 is shown, representing the neural networks discussed herein, including neural network 101 and neural network 601. Artificial neural network 1100 includes an input layer 1101, multiple hidden layers represented by hidden layers 1103 and 1105, and an output layer 1107. Input layer 1101 takes as input a feature vector encoded therein containing the direction of the sound source. Output layer 1107 generates filter design parameters as output in the form of learned modal components.

[0120] Figure 12An exemplary output layer is shown in more detail; output layer 1200 includes four components (or nodes), each producing a different output. The nodes include a low-frequency (LF) node 1210, a first peak node (peak node 1220), a Kth peak node (peak node 1230), and a high-frequency (HF) node 1240. Each node in the final hidden layer is connected to each node in output layer 1200. The hidden layer outputs to feeds into various functions within each output node, including linear activation functions, sigmoid activation functions, and linear scaling functions.

[0121] LF node 1210 includes a linear activation function 1211, a sigmoid activation function 1213, and a linear scaling function 1215. The linear activation function 1211 generates a gain value based on the hidden layer output. The sigmoid activation function 1213, combined with the linear scaling function 1215, generates a center frequency value. The gain and center frequency values ​​generated by LF node 1210 are provided as design parameters for the LF filter component 1251 of the IIR filter. These design parameter values ​​can be converted into filter coefficient values ​​for configuring the LF filter component 1251.

[0122] Peak node 1220 includes a linear activation function 1221, two sigmoid activation functions (1223 and 1227), and two corresponding linear scaling functions 1225 and 1229. Linear activation function 1221 generates a gain value based on the hidden layer output. Sigmoid activation function 1223, combined with linear scaling function 1225, generates a center frequency value. Sigmoid activation function 1227, combined with linear scaling function 1229, generates a bandwidth value. The gain, center frequency, and bandwidth values ​​generated by peak node 1220 are provided as design parameters for peak filter component 1252 of the IIR filter. These design parameter values ​​can be converted into filter coefficient values ​​for configuring peak filter component 1252.

[0123] Peak node 1230 also includes a linear activation function 1231, two sigmoid activation functions (1233 and 1237), and two corresponding linear scaling functions 1235 and 1239. Linear activation function 1231 generates a gain value based on the hidden layer output. The sigmoid activation function 1233, combined with linear scaling function 1235, generates a center frequency value. The sigmoid activation function 1237, combined with linear scaling function 1239, generates a bandwidth value. The gain, center frequency, and bandwidth values ​​generated by peak node 1230 are provided as design parameters for the peak filter component 1253 of the IIR filter. These design parameter values ​​can be converted into filter coefficient values ​​for configuring the peak filter component 1253.

[0124] HF node 1240 includes a linear activation function 1241, a sigmoid activation function 1243, and a linear scaling function 1245. Linear activation function 1241 generates a gain value based on the hidden layer output. The sigmoid activation function 1243, combined with the linear scaling function 1245, generates a center frequency value. The gain and center frequency values ​​generated by HF node 1240 are provided as design parameters for the HF filter component 1254 of the IIR filter. The design parameter values ​​can be converted into filter coefficient values ​​for configuring the HF filter component 1254.

[0125] The various implementations of the present technology discussed above provide a wide range of technical effects, advantages, and / or improvements to computing systems and components. For example, various implementations may include one or more of the following technical effects, advantages, and / or improvements: 1) unconventional and non-traditional dynamic implementations of spatial interpolation of modal components; 2) unconventional and non-traditional operations for spatial training of neural networks; 3) dynamic transformation of anechoic audio signals to spatialized audio signals; 4) unconventional and non-traditional use of object-specific parameters to train neural networks to perform spatial interpolation of modal components based on object specificity; and 5) unconventional and non-traditional use of object-specific parameters during inference to generate object-specific learned modal components. Furthermore, the lower computational complexity of IIR compared to FIR makes the disclosed interpolation techniques particularly suitable for resource-constrained environments or any setup where energy efficiency is paramount.

[0126] It is also understandable that the disclosed implementation allows the neural network to be trained from a limited dataset because the number of output values ​​is low compared to prior art. That is, existing methods output the entire amplitude-frequency response for a given HRTF, i.e., the gain value at each frequency. The amplitude response is typically approximated at thousands of different frequencies, and therefore the size of the output layer of existing methods is the number of frequencies, i.e., thousands. Since the number of available HRTFs for model training is very limited, accurately learning a neural network to output so many values ​​is difficult and impractical. Furthermore, deploying a neural network with so many output values ​​in resource-constrained environments is difficult and impractical.

[0127] Conversely, the modal methods disclosed herein produce a much smaller number of outputs, typically 30 to 100 depending on the number of peak filters (K). Since it is generally easier to accurately learn neural network models from limited data, the modal methods disclosed herein offer performance advantages in terms of speed and resource consumption compared to existing methods when the number of output values ​​is small. Furthermore, the modal components disclosed herein approximate the most important perceptual features of the impulse response, and the modeling power of neural networks is used for these features, whereas existing methods waste modeling power attempting to estimate many low-level details of the amplitude-frequency response that are generally irrelevant to perception.

[0128] Figure 13 Two methods are briefly illustrated. Figure 1301 shows a highly simplified representation of the number of data points estimated by a neural network for a given impulse response. Each data point in Figure 1301 represents the amplitude of a signal at a given frequency. Conversely, Figure 1303 illustrates the modal approach, where only thirteen values ​​(K=3) need to be estimated: three values ​​for each peak filter (center frequency (fc), bandwidth (fb), and gain (g), k=1, 2, 3); the center frequency and gain for the low-mounted filter (k=0); and the center frequency and gain for the high-mounted filter (k=4). Furthermore, the estimated modal values ​​correspond to the most important perceptual characteristics in the spectrum: peak / valley values ​​(modeled by the peak filter) and spectral roll-off (modeled by the mount filter).

[0129] Figure 14 A computing device 1401 is illustrated, representing any system or collection of systems capable of implementing the various processes, programs, services, and scenarios disclosed herein. Examples of computing device 1401 include, but are not limited to, desktop and laptop computers, tablet computers, mobile computers, audio devices, and wearable devices. Examples may also include server computers, web servers, cloud computing platforms, and data center equipment, as well as any other type of physical or virtual server machine, container, and any variation or combination thereof.

[0130] The computing device 1401 can be implemented as a single device, system, or apparatus, or it can be implemented in a distributed manner as multiple devices, systems, or apparatuses. The computing device 1401 includes, but is not limited to, a processing system 1402, a storage system 1403, software 1405, a communication interface system 1407, and a user interface system 1409. The processing system 1402 is operatively connected to the storage system 1403, the communication interface system 1407, and the user interface system 1409.

[0131] Processing system 1402 loads and executes software 1405 from storage system 1403. Software 1405 includes and implements spatial interpolation processing 1406, which represents audio processing 200, training processing 800, multi-object training processing 900, and object-specific training processing 1000. When executed by processing system 1402, software 1405 instructs processing system 1402 to operate in the manner described herein, at least for the various processing, operational scenarios, and sequences discussed in the foregoing implementation. Computing device 1401 may optionally include other devices, features, or functions not discussed for brevity.

[0132] Still refer to Figure 14The processing system 1402 may include a microprocessor and other circuitry that retrieves and executes software 1405 from storage system 1403. The processing system 1402 may be implemented within a single processing device, or it may be distributed across multiple processing devices or subsystems that cooperate with each other in executing program instructions. Examples of the processing system 1402 include general-purpose central processing units, graphics processing units, digital signal processors, special-purpose processors and logic devices, and any other type of processing device, combinations or variations thereof.

[0133] Storage system 1403 may include any computer-readable storage medium that is readable by processing system 1402 and capable of storing software 1405. Storage system 1403 may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (e.g., computer-readable instructions, data structures, program modules, or other data). Examples of storage media include random access memory, read-only memory, magnetic disks, optical disks, flash memory, virtual and non-virtual memory, magnetic tape cassettes, magnetic tapes, disk storage devices or other magnetic storage devices, or any other suitable storage medium. In any case, computer-readable storage media are not for propagating signals.

[0134] In addition to computer-readable storage media, in some implementations, storage system 1403 may also include computer-readable communication media on which at least some of the software 1405 can be transmitted internally or externally. Storage system 1403 may be implemented as a single storage device, but may also be implemented across multiple storage devices or subsystems that coexist or are distributed together. Storage system 1403 may include additional elements such as a controller capable of communicating with processing system 1402 or other possible systems.

[0135] Software 1405 (including spatial interpolation processing 1406) can be implemented as program instructions, etc., and when executed by processing system 1402, can instruct processing system 1402 to operate according to the various operating scenarios, sequences, and processes described herein. For example, software 1405 may include program instructions for implementing the inference and training processes described herein.

[0136] In particular, program instructions may include various components or modules that cooperate or interact to perform the various processing and operational scenarios described herein. These components or modules may be implemented as compiled or interpreted instructions, or some other variation or combination of instructions. The components or modules may execute synchronously or asynchronously, serially or in parallel, in a single-threaded environment or in a multi-threaded environment, or according to any other suitable execution paradigm, variation, or combination thereof. Software 1405 may include additional processing, programs, or components, such as operating system software, virtualization software, or other application software. Software 1405 may also include firmware or some other form of machine-readable processing instructions executable by processing system 1402.

[0137] Generally, when loaded into and executed by processing system 1402, software 1405 can transform a suitable device, system, or apparatus (represented by computing device 1401) from a general-purpose computing system into a specialized computing system tailored to perform inference and / or training in an optimized manner. In practice, encoding software 1405 on storage system 1403 can alter the physical structure of storage system 1403. Specific changes to the physical structure can depend on various factors in different implementations of this specification. Examples of these factors may include, but are not limited to, the technology used to implement storage system 1403, whether the computer storage media is characterized as primary or secondary memory, and other factors.

[0138] For example, if the computer-readable storage medium is implemented as a semiconductor-based memory, the software 1405 can change the physical state of the semiconductor memory when program instructions are encoded therein, such as by changing the state of the transistors, capacitors, or other discrete circuit elements constituting the semiconductor memory. Similar changes can also occur with magnetic or optical media. Other changes to the physical medium can be made without departing from the scope of this specification, and the foregoing examples are provided only for the convenience of the present discussion.

[0139] The communication interface system 1407 may include communication connections and devices that allow communication with other computing systems (not shown) via a communication network (not shown). Examples of connections and devices that allow inter-system communication may include network interface cards, antennas, power amplifiers, RF circuitry, transceivers, and other communication circuitry. Connections and devices may communicate via a communication medium to exchange communication with other computing systems or networks of systems, such as metal, glass, air, or any other suitable communication medium. The aforementioned media, connections, and devices are well-known and need not be discussed in detail herein.

[0140] Communication between computing device 1401 and other computing systems (not shown) may occur on one or more communication networks and according to various communication protocols, combinations of protocols, or variations thereof. Examples include intranets, the Internet, the Internet of Things, local area networks, wide area networks, wireless networks, wired networks, virtual networks, software-defined networks, data center buses and backplanes, or any other type of network, combination of networks, or variations thereof. The aforementioned communication networks and protocols are well known and need not be discussed in detail here.

[0141] Those skilled in the art will understand that various aspects of the present invention can be embodied as systems, methods, or computer program products. Therefore, various aspects of the present invention can take the form of a completely hardware implementation, a completely software implementation (including firmware, resident software, microcode, etc.), or an implementation combining software and hardware aspects, all of which are collectively referred to herein as “circuit,” “module,” or “system.” Furthermore, various aspects of the present invention can take the form of a computer program product embodied in one or more computer-readable media, and having computer-readable program code on said one or more computer-readable media.

[0142] In fact, the included description and figures depict specific embodiments to teach those skilled in the art how to manufacture and use the optimal mode. For the purpose of teaching the principles of the invention, some conventional aspects have been simplified or omitted. Those skilled in the art will understand that variations of these embodiments fall within the scope of this disclosure. Those skilled in the art will understand that the features described above can be combined in various ways to form multiple embodiments. Therefore, the invention is not limited to the specific embodiments described above, but is defined only by the claims and their equivalents.

Claims

1. An audio processing method, wherein, The method uses a processor linked to stored instructions implementing the method, wherein the instructions, when executed by the processor, perform the steps of the method, the method comprising the following steps: A neural network is executed to generate modal outputs based at least on a spatial input, wherein the spatial input includes a sound source direction, and wherein the modal outputs include learned modal components of an impulse response generated by the neural network based on the sound source direction; The coefficients of the infinite impulse response (IIR) filter are determined based on the learned modal components of the impulse response generated by the neural network; and The IIR filter configured with the aforementioned coefficients is used to process the silencing audio signal to generate a spatialized audio signal.

2. The audio processing method according to claim 1, wherein, The modal components of the learned impulse response include the center frequency, bandwidth, and gain.

3. The audio processing method according to claim 2, wherein, The sound source direction includes the direction of the sound source relative to the listener's position, and the impulse response includes the head-related transfer function (HRTF).

4. The audio processing method according to claim 3, wherein, The anechoic audio signal includes the sound associated with the sound source, and the method further includes configuring the IIR filter with the coefficients and outputting the spatialized audio signal to produce a directional effect of the sound at the listener's location.

5. The audio processing method according to claim 4, wherein, The method further includes the steps of: training the neural network with training data to generate modal outputs based on spatial inputs.

6. The audio processing method according to claim 5, wherein, The training data includes HRTF samples associated with the listener's location, wherein the direction of the sound source in each of the HRTF samples is different relative to each of the other HRTF samples.

7. The audio processing method according to claim 6, wherein, Training the neural network using the training data includes: for each of the HRTF samples: The direction of the sound source is provided as input to the neural network; The output is obtained from the neural network, the output including the learned modal components of the HRTF associated with the direction of the sound source; The coefficients of the IIR filter are determined based on the learned modal components; The estimated frequency domain amplitude response of the IIR filter is determined based on the coefficients; and Perform a comparison of the estimated frequency domain amplitude response of the IIR filter with the known frequency domain amplitude response of the HRTF; and The weights in the artificial neural network are updated based on the comparison.

8. The audio processing method according to claim 7, wherein, The training data also includes audience identities associated with the HRTF samples, and the input to the neural network also includes an audience identity associated with an HRTF sample.

9. A computing device, the computing device comprising: Processing circuitry, the processing circuitry being configured to at least: A neural network is executed to process a spatial input, the neural network being trained to produce a modal output, wherein the spatial input includes a sound source direction, and wherein the modal output includes a learned modal component of an impulse response associated with the sound source direction; The coefficients of the Infinite Impulse Response (IIR) filter are determined based on the learned modal components of the impulse response obtained from the neural network; and The audio signal is processed using the IIR filter, the IIR filter being configured with coefficients for increasing the spatialization of the audio signal; and An audio circuit configured to output the audio signal.

10. The computing device according to claim 9, wherein, The sound source direction includes the direction of the sound source relative to the listener's position, and the impulse response includes a head-related transfer function (HRTF) modeled by the neural network for the listener's position with respect to the direction of the sound source.

11. The computing device according to claim 10, wherein, The audio signal includes sound associated with the sound source, and the processing circuitry is further configured to program the IR filter with the coefficients.

12. The computing device according to claim 11, wherein, The neural network is trained using training data to generate modal outputs based on spatial inputs.

13. The computing device according to claim 12, wherein, The training data includes HRTF samples associated with the audience locations.

14. The computing device according to claim 13, wherein, The direction of the sound source in each of the HRTF samples is different relative to each of the other HRTF samples.

15. The computing device according to claim 14, wherein, The training data also includes audience identities associated with the HRTF samples, and the spatial input also includes audience identities.

16. The computing device according to claim 9, wherein, The IIR filter includes a cascaded IIR filter having multiple IIR filter sections, wherein the multiple IIR filter sections include a low-frequency (LF) section, a peak frequency (PF) section, and a high-frequency (HF) section.

17. The computing device according to claim 16, wherein, The modal components of the learned impulse response include: a) the center frequency and gain of the LF portion; b) the center frequency, bandwidth, and gain of the PF portion; and c) the center frequency and gain of the HF portion.

18. One or more computer-readable storage media storing program instructions that, when executed by one or more processors of a computing device, instruct the computing device to at least: A spatial input is provided to a neural network, which generates a modal output based on the spatial input, wherein... The spatial input includes the direction of the sound source, and the modal output includes learned modal components of the impulse response generated by the neural network based on the direction of the sound source. The coefficients of the infinite impulse response (IIR) filter are determined based on the learned modal components of the impulse response generated by the neural network; and Configure the IIR filter based on the coefficients.

19. One or more computer-readable storage media according to claim 18, wherein, The program instructions also instruct the computing device to process the silencing audio signal using the IIR filter configured with the coefficients to generate a spatialized audio signal.

20. The computer-readable storage medium according to claim 19, wherein, The sound source direction includes the direction of the sound source relative to the listener's position, and the impulse response includes a head-related transfer function (HRTF) modeled by the neural network for the listener's position with respect to the direction of the sound source.

21. A method for training an artificial neural network, the method comprising the following steps: Spatial features are extracted from impulse response samples, wherein each of the impulse response samples includes spatial features and an associated impulse response; For each impulse response sample in the impulse response samples: The artificial neural network is provided with feature vectors as input; The output is obtained from the artificial neural network, the output including the learned modal components of the impulse response; The coefficients of the impulse response (IR) filter are determined based on the learned modal components of the impulse response; The estimated frequency domain amplitude response of the IR filter is determined based on the coefficients; and Perform a comparison between the estimated frequency domain amplitude response of the IR filter and the known frequency domain amplitude response of the impulse response; and The weights in the artificial neural network are updated based on the comparison.

22. The method according to claim 21, wherein, The IR filter includes an infinite impulse response (IIR) filter, and the learned modal components include center frequency, bandwidth, and gain.

23. The method according to claim 22, wherein, The impulse response includes either the head-related transfer function (HRTF) or the room impulse response (RIR).

Images