Method and system for generating acoustic impulse responses for 3D room models using a hybrid wave-based solver and a geometric acoustics-based solver

Abstract

Disclosed is a computer-implemented method for generating an impulse response for a listening point in a room, the method including receiving a 3D model of the room, the location of at least one sound source in the 3D model of the room, and acoustic characteristics of at least one boundary in the 3D model of the room, using a wave-based solver to determine a wave-based impulse response for wave-based propagation of impulses emitted from the at least one sound source in the 3D model of the room and received at the listening point within a first acoustic frequency range, using a geometric acoustics-based solver to determine a geometric impulse response for ray-based propagation of impulses emitted from the at least one sound source in the 3D model of the room and received at the listening point within a second acoustic frequency range, and generating the impulse response by merging the wave impulse response and the geometric impulse response. Methods and systems using the disclosed computer-implemented method and features thereof are also disclosed.

Description

[Technical Field]

[0001] This specification relates to the simulation of acoustic wave propagation, and the rendering and use of the simulation. [Background technology]

[0002] The theory of acoustic wave propagation is well established: acoustic waves emanating from a source and propagating through some n-dimensional space-time domain can be mathematically described using wave equations based on the fundamental physical principles of conservation of mass and momentum.

[0003] However, the application of computer systems and computer-implemented methods to acoustic simulation requires that acoustic theory be translated into computationally feasible simulation methods.

[0004] Many different simulation methods are known in the art, such as the finite-difference time-domain method (FDTD), boundary element method (BEM), or finite volume method (FVM), and what they have in common is that each is used with a consideration of speed and accuracy: if a fast simulation is desired, the trade-off is less accuracy (low fidelity), and if an accurate simulation (high fidelity) is desired, the simulation can often require several days to run.

[0005] Advances in ever greater processing power, both local and cloud-based, can improve both the speed and fidelity of simulations. However, in many cases, simply using conventional, common simulation methods with increased processing power does not produce satisfactory results. This may be because, for example, known simulation methods are not optimized for the architecture and implementation of future computer processing systems.

[0006] For example,

[17] describes many simulation methods in the field of room acoustics simulation, which is incorporated herein by reference.

[0007] Therefore, there is a need to further develop and improve acoustic simulation methods. Summary of the Invention

[0008] This disclosure relates to the implementation and optimization of three types of simulation methods: a geometric acoustic solver, a wave-based solver, and a hybridization or combination of the two simulation methods. Additionally, uses and applications involving the simulation are described.

[0009] Geometric acoustic solver Geometric acoustic solvers are commonly known for their fast but low-accuracy simulations.

[0010] The basic principle of the geometric acoustic solver is to adopt the high frequency assumption, where wavelengths are assumed to be vanishingly small compared to the dimensions of the room and the obstacles within it. The concept of a sound wave is then replaced by that of a bundle of sound rays, where each ray represents a small portion of the wavefront that vanishes through an aperture. The rays propagate through the room according to the laws of ray optics.

[0011] Examples of common geometric acoustic methods used in geometric acoustic solvers are ray tracing and image source methods

[19] . Improvements to geometric acoustic solvers that apply ray tracing and image source methods are further described herein.

[0012] The geometric acoustic approximation is generally considered acceptable at high frequencies in large rooms where the dimensions and size of obstacles are orders of magnitude larger than the acoustic wavelength. When obstacles within the room or the dimensions of the room become comparable in size to the wavelength, the approximation becomes inadequate and inaccurate.

[0013] Wave-based solver Wave-based solvers are generally known for their slow but highly accurate simulations.

[0014] The wave-based methods used in such solvers employ numerical techniques to directly solve the governing partial differential equations that describe the wave motion in a fictitious domain, such as that representing an air volume. This can be, for example, the wave equation in the time domain or the Helmholtz equation in the frequency domain.

[0015] Thus, the concept of various wave-based methods is to divide (discretize) a virtual domain of interest into smaller subdomains and solve algebraic equations in each subdomain. Thus, the wave-based methods disclosed herein can be understood as methods for solving partial differential equations using discretization techniques.

[0016] As discussed herein, a wave-based solver includes or refers to a finite element method (FEM), which may be, for example, a nodal finite element method, which encompasses numerical methods such as the discontinuous Galerkin finite element method (DG / DGFEM) or the spectral element method (SEM).

[0017] In particular, the discontinuous Galerkin finite element method (DG / DGFEM scheme) has been shown to be efficient in simulating acoustic wave propagation as discussed herein and is considered below.

[0018] To account for the DG scheme, the wave-based engine solves the following system of differential equations that describe wave propagation in a lossless air medium [1]:

number

number

number

number

number

number

number

number

number

number

number

number

[0019] As discussed further herein, the spatial domains Ω are non-overlapping and their potentially unstructured mesh elements or sub-domains Ω e In volumetric meshing, the geometry is divided or discretized into a set of elements. The elements can be of any shape, e.g., tetrahedrons, hexahedrons, pyramids, etc., or a combination of different shapes. However, a goal of uniformity of the mesh elements with respect to shape and element size across the mesh can improve certain processes. Thus, in some embodiments, it may be preferable not to deform elements of the same shape, e.g., tetrahedrons.

[0020] Next, the variational formulation of the propagation equation is

number

number

number

number

number

number

number

number

number

number

[0021] To implement data exchange between elements, upwind numerical fluxes can be used. Upwind numerical fluxes can have optimal dispersion properties and can be interpreted by considering the propagation direction of characteristic waves at element interfaces. Due to the hyperbolic nature of the propagation equation (3), the eigendecomposition of the usual flux matrix is

number

number

number

number

[0022] The upwind flux is the wave exiting the element. e On the other hand, the wave entering the element should depend only on the external solution.

number

number

number

number

number

number

number

number

number

number

[0023] As detailed in [8], the approximate solution of the strong formulation [Eq. (4)] is given by

number

number

number

number

number

number

number

number

[0024] Interpolating Polynomial

number

number

[0025] Inserting equation (9) into equation (4) results in the following semi-discrete system:

number

number

number

number

number

number

number

number

number

number

number

number

[15] .

[0026] Thus, generally speaking, the derivation of the SEM / DG / DGFEM scheme, in one embodiment as described above, involves four steps: the governing equation, equation (3), can be formulated in a modified formulation according to equation (4). Continuity across the virtual domain, discretized into subdomains, is ensured using the numerical fluxes shown in equation (5). Finally, the problem can then be formulated as a numerical semi-discrete system according to equation (10). This effectively means that the spatial derivatives found in the governing equations have been converted into simple algebraic equations that can be solved by a computer. All that remains deals with the time derivatives on the left side of equation (10), which are handled by a time-marching algorithm as discussed herein.

[0027] In one aspect, the present disclosure relates to a computer-implemented method for acoustic simulation in a virtual domain, the computer-implemented method including: obtaining input data including at least one model of the virtual domain, at least one sound source in the virtual domain, and at least one acoustic characteristic of the virtual domain; obtaining a wave-based solver for determining wave-based propagation of sound in the virtual domain based on the at least one sound source and the at least one acoustic characteristic in a first acoustic frequency range; and obtaining a geometric acoustic-based solver for determining ray-based propagation of sound in the virtual domain based on the at least one sound source and the at least one acoustic characteristic in a second acoustic frequency range, and outputting the wave-based propagation and the geometric acoustic-based propagation.

[0028] In one embodiment, a computer-implemented method for generating an impulse response of a listening point in a room may be provided, the method comprising: (a) receiving a 3D model of a room, a location of at least one sound source in the 3D model of the room, and acoustic characteristics of at least one boundary in the 3D model of the room; (b) using a wave-based solver to determine a wave-based impulse response of a wave-based propagation of an impulse emitted from at least one sound source within the 3D model of the room and received at a listening point within a first acoustic frequency range; (c) using a geometric acoustics-based solver to determine a geometric impulse response of a ray-based propagation of an impulse emitted from at least one sound source within the 3D model of the room and received at a listening point within a second acoustic frequency range; (d) generating an impulse response by merging the wave impulse response and the geometric impulse response.

[0029] For example, as discussed herein, when a sound source emits an impulse, an impulse response is simulated or generated. Thus, sound propagation describes the impulse response as it propagates within a room.

[0030] As described further herein, the output can be in many different forms and embodiments, for example, it can be a visual rendering, it can be data for use in further processing, for example to train a neural network, or it may be used to provide an audio rendering of a listening experience in a virtual environment.

[0031] In another aspect, a wave-based solver for simulating sound propagation within at least one model of a virtual domain is disclosed based on at least one sound source within the virtual domain and at least one acoustic property of the virtual domain. The at least one acoustic property and the at least one sound source within the virtual domain may also be similar to those described further. The wave-based solver may include a finite element method (FEM), such as a discontinuous Galerkin finite element method (DGFEM) or a spectral element method (SEM).

[0032] In another aspect, a wave-based method for simulating sound propagation is disclosed, the method including one or more of the steps of executing the features and modules discussed herein with respect to a wave-based solver. The wave-based method can be implemented, for example, on a computer.

[0033] In a further aspect, a geometric acoustic solver is disclosed for simulating sound propagation within at least one model of a virtual domain based on at least one sound source within the virtual domain and at least one acoustic property within the virtual domain, the geometric acoustic solver comprising: a) at least one image source module for determining at least one image source simulation by simulating sound propagation; b) at least a first acoustic ray tracing module for determining at least a first ray tracing simulation by simulating the propagation of sound; c) a hybridization module for combining at least a first ray tracing simulation and at least one image source simulation into a geometric acoustic simulation of sound propagation; d) an output module for preparing an output of the geometric acoustic simulation.

[0034] In another aspect, a geometric acoustics method for simulating sound propagation is disclosed, the method including one or more of the steps of implementing the features and modules discussed herein with respect to a geometric acoustics solver. The geometric acoustics method can be implemented, for example, on a computer.

[0035] Wave-based simulation and geometric acoustic-based simulation may be improved and hybridized as discussed herein. Furthermore, applications of simulation, such as rendering and use of acoustic simulation, may benefit from various kinds of improvements. The acoustic simulation disclosed herein may even enable different types of applications that were not previously possible.

[0036] Acoustic simulation, simulation of sound propagation, or sound propagation simulation may generally be understood as simulation of acoustic waves. Acoustic waves are a type of energy propagation through a medium by adiabatic loading or unloading. Acoustic waves may include audible sound(s), seismic waves, or ultrasonic waves(s). Applications of the methods and systems described herein preferably fall within the range of acoustic waves, more preferably within the range of audible sound(s). Acoustic waves may also be defined herein as acoustic signal(s).

[0037] In another aspect, a system for generating an impulse response for a listening point in a room is disclosed, the system comprising: The computer system includes a processor coupled to a memory, the processor comprising: receiving a 3D model of a room, a location of at least one sound source within the room, and acoustic characteristics of at least one boundary within the 3D model of the room; determining, using a wave-based solver, a wave-based impulse response for wave-based propagation of an impulse emitted by at least one sound source within the 3D model of the room and received at a listening point within a first acoustic frequency range; determining, using a geometric acoustics-based solver, a geometric impulse response of a ray-based propagation of an impulse emitted by at least one sound source within the 3D model of the room and received at a listening point within a second acoustic frequency range; The impulse response is configured to be generated by merging a wave impulse response and a geometric impulse response.

[0038] The systems disclosed above may be configured to perform a computer-implemented method for generating impulse responses for listening points in a room as disclosed herein.

[0039] In one application disclosed herein, the acoustic simulation can be used in a method for training a machine learning-driven audio algorithm. obtaining input data including at least one model of a virtual domain, a sound source located in the at least one model in the virtual domain, and at least one acoustic characteristic of the at least one model of the virtual domain; performing at least one sound propagation simulation in the virtual space from at least one sound source to at least one sound receiver located in at least one model of the virtual domain; obtaining at least one sound propagation simulation output from the at least one sound propagation simulation; training a machine learning model for a machine learning driven audio device with the at least one sound propagation simulation output.

[0040] The steps of the disclosed method may be performed in any order that benefits the method. The at least one sound propagation simulation output may vary depending on the sound propagation simulation used. It may preferably take any form suitable for training a machine learning model for a machine learning-driven audio algorithm or for further processing by at least one secondary system before being used to train a machine learning model for a machine learning-driven audio device. The machine learning-driven audio algorithms described herein may be used for various applications, such as blind room acoustic modeling, dereverberation, echo cancellation, speech recognition, beamforming, or virtual sound rendering.

[0041] For example, in one additional aspect, a method may be provided for providing a machine learning model for audio compensation in an audio device, the audio device including at least one microphone, the method including receiving an audio signal at the at least one microphone and generating a compensated audio signal by using the machine learning model for audio compensation.

[0042] The method can be trained, for example, by a computer-implemented method, the method including: receiving 3D models of a plurality of rooms, each of the 3D models including at least one sound source and at least one acoustic characteristic; receiving a plurality of impulse responses at listening positions in each of the plurality of rooms; and training a machine learning model for audio compensation using at least the plurality of impulse responses as inputs for training the machine learning model. Training the machine learning model for audio compensation may include using at least the plurality of impulse responses as inputs.

[0043] In other words, acoustic simulations can be used as described herein to generate synthetic data, such as generated impulse responses, which can be used to train machine learning-driven audio algorithms in a fast and accurate manner, especially when compared to current methods in which data is generated in real-world conditions, which complicate generating relatively large amounts of data in a relatively short amount of time. For example, the use of synthetic data for audio compensation as described herein reduces the costly and time-consuming task of performing physical measurements and recordings.

[0044] In one aspect, an audio device may be provided that includes at least one microphone, the audio device including a processing system for applying machine learning-based audio compensation as described herein, the audio device configured to receive an audio signal with the at least one microphone and apply the machine learning-based audio compensation to the audio signal to generate a compensated audio signal.

[0045] The audio device may be, for example, an audio device for a web-based conference. Thus, when a speaker's audio is received at a first audio device and forwarded to a remote audio device, the sound is processed using audio compensation to ensure that the audio is played properly and clearly through the remote audio device.

[0046] In one aspect, a system for training a machine learning model for audio compensation is described, the system comprising: The computer system includes a processor coupled to a memory, the processor comprising: receiving 3D models of a plurality of rooms, each of the 3D models including at least one sound source and at least one acoustic characteristic; receiving multiple impulse responses at listening positions in multiple rooms; A machine learning model for audio compensation is configured to be trained using at least the plurality of impulse responses as input.

[0047] The systems disclosed above may be configured to perform a computer-implemented method for training a machine learning model for audio compensation as disclosed herein.

[0048] In another aspect, a method for determining a head-related transfer function is disclosed, the method comprising: obtaining a user's head geometry; performing a simulation of sound propagation from at least one audio source to a geometry of a user's head (typically a geometry proximate to the user's ear, into the user's ear or eardrum), wherein the simulation of sound propagation is based on a sound propagation simulation using a wave-based solver; and determining a head-related transfer function (HRTF) of the user's head based on a simulation of sound propagation.

[0049] In a further aspect thereof, there may be provided a computer implemented method for determining a head-related transfer function, the method comprising: (a) receiving a 3D model of a user's head and a location of at least one sound source in the 3D model representing an eardrum or a proximate thereto; (b) using a wave-based solver to determine a plurality of wave-based impulse responses from an impulse emitted by at least one sound source, the plurality of wave-based impulse responses being determined in a plurality of digital representations of a head-mounted receiver; (c) determining a head-related transfer function (HRTF) of the user's head based on the plurality of wave-based impulse responses determined from the plurality of digital representations of the head receivers.

[0050] The geometry of the user's head can be obtained by multiple methods. The geometry can be obtained by 3D scanning the geometry of the user's head or the shape of at least some or all of the outer ear(s). One method can consist of capturing multiple images of the user's head and using the multiple images to construct the geometry of the user's head. Another method can consist of using a depth camera, whereby the depth camera can capture an image of the user including the user's head. Depth information is captured along with the multiple images, thus enabling a system, such as a computer, a computer-implemented method, or any computer-based system, to construct the geometry of the user's head. Geometry can also be obtained from an auxiliary interface, where the geometry of the user's head is ready for use by the method.

[0051] The head-related function, also known as the anatomical transfer function (ATF), is a response that characterizes how the ear receives sound from a point in space. When sound strikes a listener, the size and shape of the head, ears, ear canals, head density, and the size and shape of the nasal and oral cavities can transform the sound and affect how it is perceived. Some frequencies in the sound spectrum may be boosted, while others may be attenuated. HRTFs can vary greatly from person to person.

[0052] HRTFs can be used, for example, to personalize a user's audio experience: because HRTFs characterize how a user normally receives sound, personalized HRTFs can provide a highly realistic experience for users when applied, for example, in headsets, car audio, home entertainment, games, virtual reality, etc.

[0053] By combining a person's orientation with their HRTFs, for example, so that certain filters are applied to describe sounds coming from the front and other filters are applied for sounds coming from the side or rear, the acoustic experience becomes very realistic. Furthermore, different filters may be required for different frequencies and sound levels.

[0054] Thus, in one embodiment, a device for providing auralization, such as a headphone set for providing binaural auralization, may be configured to apply the HRTFs disclosed herein to an audio signal played to a user.

[0055] Using simulation of sound propagation to determine the HRTFs for a user's head provides a reliable, fast and efficient method.

[0056] In a further aspect, systems for implementing the described methods are also provided herein.

[0057] Thus, in one embodiment, a system is disclosed for simulating the propagation of sound within at least one model of a virtual domain using a wave-based solver, a geometric acoustic solver, or a combination thereof, as described herein.

[0058] The system includes a storage medium for storing data and a computer program describing the method of simulating sound propagation described herein, a processor for executing the method of simulating sound propagation described herein, at least one input device for inputting data into the system, and an output device for outputting data from the system.

[0059] For example, in one embodiment, the storage medium and / or processor are cloud-based, and data is entered into the system from a personal computer via a keyboard and monitor, e.g., by sending a CAD model of the virtual space over the internet to a cloud service and loading it into the system.

[0060] The output may also be provided through a personal computer via the cloud service, for example by presenting the output on a monitor, rendering the output on a monitor, or playing audio output through the computer's speakers, external speakers, a set of VR (virtual reality) glasses, and / or an audio headset. DETAILED DESCRIPTION OF THE INVENTION

[0061] The term "using" as used throughout this patent application includes the acts of "performing" or "applying." In the context of this patent application, "using" which refers to the performance of a specified act or operation may include running or applying a particular process.

[0062] The virtual domain may be a one-dimensional (1D) domain, a two-dimensional (2D) domain, or a three-dimensional (3D) domain, such as a three-dimensional model (3D) of a virtual space.

[0063] For example, a one-dimensional (1D) domain can be used to simplify simulations when performing simulation analysis of the absorption properties of materials. In this case, wave propagation can be reduced to one dimension, thereby reducing wave propagation to planar wave propagation instead of spherical wave propagation. This makes the simulation of acoustic waves much faster.

[0064] A two-dimensional (2D) domain may have applications where acoustic wave propagation is simulated in an outdoor environment. A 2D domain may also be used if the 3D domain can be cut into slices. Solving a 2D domain with a sound propagation simulation may be less complex than solving a three-dimensional (3D) domain. It may also be useful to simulate sound propagation at a fixed height plane within a 3D domain, where sound propagation may be relevant for analysis at a specific plane, such as within a car cabin. Simulation in a 1D or 2D domain reduces the complexity of the sound propagation simulation by removing one or two dimensions from the equations described herein, such as Equation (1) and / or Equation (2).

[0065] The three-dimensional domain (3D) of the virtual space can be implemented as a 3D model of the virtual space, which can be an enclosed air volume, but can also be a semi-open air volume, or an open space without boundaries.

[0066] The 3D model of the virtual space can be implemented or obtained thanks to existing computer-implemented modeling methods, such as a 3D model builder. The 3D model can be closed. The 3D model can also emulate a semi-open space by setting boundaries far from one another, where the amplitude of the acoustic signal can be so low that the effect of the boundaries on the acoustic signal is almost negligible. The boundaries can also be defined with very high absorption coefficients, thereby essentially simulating sound leaking into an open environment. The acoustic properties of the 3D model can be the acoustic properties of materials that can be part of the 3D model. For example, materials can have acoustic properties such as surface impedance, which can characterize how a material can react when an acoustic signal or wave impinges on its surface. At least one sound source can be installed or arranged within the 3D model. The at least one sound source can be configured at a specific location where an acoustic signal can be transmitted within the 3D model. The at least one sound source can be oriented in a specific orientation and have specific frequency-dependent directional characteristics, radiating sound in different directions with different intensities. The particular orientation angle can be defined by a particular orientation of the at least one sound source. The at least one sound source can also be arranged outside the 3D model.

[0067] The wavelength-based solver can be any type of wavelength-based solver that is useful for acoustic simulation. The finite element method may be the most commonly used method that may be useful for acoustic simulation. Geometric acoustics is a branch of acoustics that studies the propagation of sound based on the concept of rays, which are considered as lines along which acoustic energy is transmitted.

[0068] The output resulting from the steps of outputting the wave-based propagation and the geometric acoustic-based propagation can have any type of format. The type of format can be a text file containing different information about the propagation of the acoustic signal within the 3D model, or a graphical interface showing the propagation and / or reflection of the acoustic signal within the 3D model. More preferably, the format of the output can match the needs of a system or user using any of the computer-implemented methods of the present disclosure.

[0069] In one embodiment, the first and second acoustic frequency ranges are different, e.g., two adjacent frequency ranges. This allows for optimal simulation using a combination of wave-based simulation and geometric acoustic simulation when considering simulation speed versus accuracy. For example, if a fast simulation is desired, the second acoustic frequency range may be larger, while if more accuracy is desired, the first acoustic frequency range may be larger.

[0070] The frequency range of interest may be the audio frequency range, which is generally between 20 Hz and 20 kHz. Within the audio frequency range, low frequencies are typically considered to fall within the range of 20 Hz to 200 Hz, mid frequencies are typically considered to fall within the range of 200 Hz to 2000 Hz, and high frequencies are typically considered to fall within the range of 2000 Hz to 20 kHz. Thus, a first audio frequency range may be, for example, a low and mid audio frequency range, e.g., 20 Hz to 2000 Hz, and a second audio frequency range may be an adjacent high audio frequency range, e.g., 2000 Hz to 20 kHz.

[0071] The first and second acoustic frequency ranges can be defined according to the geometry of the 3D model. Depending on the acoustic signal spectrum, a wave-based solver can be more accurate than a geometric acoustic-based solver. Generally, when simulating the low- to mid-frequency range of the acoustic signal, a wavelength-based solver can be appropriate, while when simulating the high-frequency range of the acoustic signal, a geometric acoustic simulation can be sufficient. The wavelength can generally define the frequency range of interest depending on the geometry of the 3D model. When wave propagation occurs in air, a frequency of 20 Hz has a wavelength of approximately 17.2 m, while a frequency of 20 kHz has a wavelength of approximately 17.2 mm. It may be noticeable that there is a factor of approximately 1000 between the high-frequency and low-frequency wavelengths. This may lead to some physical behaviors that a geometric acoustic-based solver cannot calculate / simulate with high accuracy / precision, such as interference, scattering, or potential diffraction of acoustic waves, especially in the low- to mid-frequency range. By combining both approaches, i.e., a wave-based solver and a geometric acoustic-based solver, better simulation accuracy can be achieved.

[0072] In one embodiment, the first and second acoustic frequency ranges can partially overlap, which may be desirable to obtain a smooth transition when merging the two simulation outputs. The overlap of the first and second acoustic frequencies may also depend on the convergence of the simulation.

[0073] The first and second acoustic frequency ranges may, in some embodiments, completely overlap, such that, for example, a fast initial simulation provided by a geometric acoustic solver may be desirable for evaluation, while a subsequent slower, more accurate simulation may be provided by a wave-based solver.

[0074] The lowest frequency of the second frequency range or the highest frequency of the first frequency range can be manually selected by the user. This selection provides the computer-implemented method with a frequency range in which the wave-based solver and the geometric acoustics-based solver should be used. This selection can be performed depending on the size of the 3D model. For example, a relatively small 3D model would benefit from having a relatively wide low-frequency range in order to run the wave-based solver over a relatively wide frequency range.

[0075] The lowest frequency of the second frequency range or the highest frequency of the first frequency range can be automatically selected by a computer-implemented method. The method can evaluate a frequency range where strong acoustic wave phenomena, such as diffraction and interference, occur and set the highest frequency range of the wave-based solver to cover this frequency range. This can be based, for example, on estimating the Schroeder frequency of the 3D domain of the problem, or in some cases, more advanced techniques can be used to estimate diffraction based on the surface areas that make up the 3D domain and its interior fixtures.

[0076] In one embodiment, wave-based propagation and geometric acoustic-based propagation are merged together.

[0077] For example, as described below, the output of the wave-based sound simulation and the output of the geometric-acoustic-based sound simulation may be in the format of a wave-based impulse response and a geometric-acoustic-based impulse response, respectively.

[0078] Merging and hybridizing wave-based and geometric impulse responses requires careful consideration to avoid introducing non-physical artifacts into the combined broadband solution that forms the simulated impulse response of the simulated virtual space.

[0079] The simulated wave-based propagation and the simulated geometric acoustic-based propagation can be simulated in a first frequency range and a second frequency range, respectively. The first frequency range and the second frequency range can partially overlap or completely overlap. A transition frequency is defined as the frequency at which the first frequency range and the second frequency range overlap. The transition frequency is defined as the center frequency of the overlap between the first frequency range and the second frequency range.

[0080] The transition frequency may preferably be used by the wave-based solver to set the frequency content of at least one sound source in the 3D model. The frequency content of the at least one sound source in the 3D model may include sufficient acoustic energy up to the transition frequency to have a sufficient signal-to-noise ratio. If the signal-to-noise ratio is too low, the wave-based impulse response may prove inaccurate or may produce inaccurate results at some frequencies included in the first frequency range, preferably up to the transition frequency.

[0081] The transition frequency may be a default transition frequency. The default transition frequency may be proportional to the square root of the volume of the 3D model. The default transition frequency may be based on a simplified calculation of the Schroeder frequency, assuming a reverberation time of 1 second and multiplying it by a coefficient. The reverberation time may be assumed to be 1 second. The coefficient may be comprised between 0.5 and 2, preferably between 0.1 and 5, and more preferably between 0.1 and 10. The coefficient may be selected depending on the volume of the 3D model of the room. The reverberation time may be selected as a compromise between simulation time and simulation accuracy. A short reverberation time may produce inaccurate results, thereby generating inaccurate generated impulse responses, while a long reverberation time may significantly increase simulation time without increasing the accuracy of the generated impulse responses.

[0082] The first frequency range and the second frequency range may also be referred to as a first acoustic frequency range and a second acoustic frequency range, respectively.

[0083] Thus, the step of merging the wave-based impulse response and the geometric-based impulse response may further include calibrating the sound power of at least one sound source in the 3D model so that it matches in both the wave-based solver and the geometric-based impulse response.

[0084] Calibration can be performed by estimating the sound level emitted from each sound source at 1 meter in free-field conditions, for example, within a frequency range close to the transition frequency between the first and second acoustic frequency ranges. Pressure levels can also be estimated at other distances, such as 0.5 meters, 2 meters, and / or 3 meters. The distance can be between 0.5 meters and the extent of the virtual space or 3D model of the room in which the simulation is performed.

[0085] The merging step may further include the boundary conditions being consistent for the wave-based solver and the geometric acoustic solver: if the boundary inputs are not aligned, there is a risk that artifacts such as phase mismatch and / or undesired results may occur in the transition frequency region.

[0086] Preferably, the sound pressure level radiated from each sound source at 1 meter may be 94 dB sound pressure level (SPL). In both the wave-based solver and the geometric acoustic solver, the step of calibrating the sound power of at least one sound source in the 3D model so that each sound source is consistent may be performed independently in a source correction step or calibration step included in the wave-based solver and / or the geometric acoustic solver. More preferably, the calibration step may be performed independently in the image source simulation and / or the ray tracing simulation.

[0087] The ray-based propagation described herein may be described, for example, by ray tracing methods and / or image source methods.

[0088] Advantageously, the wave-based impulse response can be digitally filtered with a low-pass filter having a cutoff frequency located substantially at the transition frequency, thereby generating a filtered wave-based impulse response. This can be performed before performing the step of calibrating the sound power. Preferably, the low-pass filter is a fourth-order Butterworth low-pass filter. A Butterworth filter is a type of signal processing filter designed to have as flat a frequency response as possible within its passband. Flatness of the frequency response in the passband can be understood in the context of signal processing, thus minimizing distortion and / or preserving the shape of the impulse response. Preferably, for the acoustic simulation discussed herein, the flat passband response of a Butterworth filter can be desirable because it can help avoid sound coloration. Uneven attenuation within the passband can change the timbre or tone of the audio signal, potentially leading to a reduction in sound fidelity. The order of the filter can be a compromise between having a sharp cutoff frequency while avoiding ripple in the time domain. The inventors have discovered that fourth order can be a sufficient order to achieve the desired filtering.

[0089] Both the wavelength-based solver and the geometric acoustic solver output both mono and spatial Ambisonics impulse responses for each listening point in the 3D model, along with acoustic parameters derived from the mentioned impulse responses.

[0090] The merging step can be digitally filtered with a high-pass filter having a cutoff frequency located substantially at the transition frequency, thereby generating a filtered geometric acoustic impulse response. This can be performed before or after performing the sound power calibrating step. Preferably, the sound power calibrating step can be performed before digitally filtering the geometric acoustic impulse response with a high-pass filter having a cutoff frequency located substantially at the transition frequency. Preferably, the high-pass filter is a fourth-order Butterworth high-pass filter.

[0091] The merging step can sum both the filtered wave impulse response and the filtered geometric acoustic impulse response, thereby generating a hybridized impulse response, where the hybridized impulse response is a combination of the filtered wave impulse response and the filtered geometric acoustic impulse response. The hybridized impulse response can be an impulse response described herein, where the impulse response can combine a wave impulse response and a geometric impulse response.

[0092] The merging step may be performed independently for all channels contained in the spatial Ambisonics impulse response.

[0093] Typically, wave-based solvers rely on complex surface impedances for their boundary conditions, while geometric acoustic solvers rely on energy-based absorption coefficients. Because these are not one-to-one comparable, one or both must be transformed into a consistent solution. In one embodiment, the geometric acoustic solver relies on impedances for its boundary conditions, thereby accounting for both energy reduction and phase shift at the boundary. As an example, this can be implemented by tracking the phase shift introduced at the boundary of every reflected ray or every generated image source, and accumulating the total phase shift that occurs as the acoustic wave travels between the source and receiver. Additionally, it may mean that the phase of the wave must be tracked for each ray extent.

[0094] Furthermore, the merging or hybridization step involves ensuring that the time arrival, i.e., phase, of the sounds propagated in the virtual space arrive at the receiver location simultaneously. In one embodiment, this is provided by using a pressure-based image source method in a geometric acoustics-based solver that models the phase shifts at the boundary conditions described above.

[0095] In one embodiment, the computer-implemented method may further include an auto-stop, which may periodically calculate the energy attenuation of sound energy in the 3D model of the room. By periodically calculating the energy attenuation of sound energy while the simulation is running, the computer-implemented method may stop the simulation when a sufficient amount of energy attenuation is achieved. For example, the simulation may stop when the ratio of the sound level of the source to the simulated sound energy reaches a predetermined threshold, e.g., between -20 dB and -80 dB, such as -20 dB, -40 dB, 60 dB, or -80 dB. The periodic calculation may be performed at each listening point or receiver arranged in the 3D model of the room, and acoustic waves or sounds are simulated. The auto-stop may stop the simulation when all receivers or listening points achieve a predefined energy attenuation value, such as -60 dB. By providing the auto-stop, the simulation can be stopped at the point where the user desires a desired energy attenuation, thereby avoiding running the simulation longer than necessary.

[0096] The wave-based solver may also include a DG module for determining the DG simulation, preferably by simulating the propagation of sound, and an output module for preparing the output of the DG simulation. The DG module may further include additional modules that may provide additional features and / or settings to the wave-based solver.

[0097] A module, such as a DG module, may be considered a module of software. It may also be considered a portion or section of software that provides the functionality of a solver. Software modules are generally considered to be separate packages or distinct parts of functionality. This is used to make the software easier to use for a particular use case or to define where boundaries exist within a program. The module(s) described herein should not be limited to being construed as software modules, but may relate to steps, functions, or portions of a solver that are not independent of each other, but rather may be interconnected, as shown in FIG. 1.

[0098] The 3D model of the virtual space may define multiple bounding surfaces of the air volume. The air volume may be semi-sealed or partially sealed, or may be completely sealed or sealed, such as a room without an opening. A semi-sealed air volume may be, for example, a room with an open window, or an open or half-open door. The air volume may be an outdoor area whose boundaries may be defined by trees, natural or artificial reliefs, or objects. The 3D model of the virtual space may be implemented / obtained through any existing modeling computer-implemented method(s), such as 3D computer graphics and / or computer-aided design application software. Some 3D computer-aided design application software, such as Rhinoceros3D, Revit, or SolidWorks, may be ideal for designing 3D models of virtual spaces. The bounding surfaces of the 3D model may be of any type, such as flat or curved. They may be a combination of both flat and curved bounding surfaces. The bounding surfaces may define the edges of the 3D model. The 3D model may be a sealed air volume. The open model can be defined by setting a boundary surface at a distance such that the amplitude of the acoustic signal becomes negligible when it reaches the boundary surface. The distance can be emulated by creating an absorbing boundary surface that can preferably absorb a large amount of acoustic wave energy, more preferably the entirety of the acoustic wave energy. In this context, the acoustic wave is not reflected back into the 3D model and is then considered in the simulation to be outside the simulated environment, i.e., the 3D model.

[0099] The boundary surface may be defined by a 2D mesh. The 2D mesh may be defined as a 2D surface mesh. The 2D mesh describes the geometry of the boundary surface. The boundary surface defines the boundary of a 3D model of the virtual domain. The 3D model of the virtual domain may be represented by a 3D volumetric mesh model. A mesh is a subdivision of a continuous geometric space into discrete geometric and topological cells. Mesh cells are typically used as discrete local neighbors of a larger domain. A preferred procedure may include dividing the boundary surface into a mesh of cells or a grid of points; for example, generating a mesh can capture the 3D model geometry using relatively high-quality cells without requiring an excessively large number of cells, which may make subsequent calculations of the acoustic simulation unwieldy and / or inefficient. The 3D mesh model may be generated in different ways depending on the complexity of the boundary surface and the accuracy required by the application and / or user.

[0100] A 3D mesh model can include curvilinear mesh elements, which can reduce geometric distortions that occur when meshing curved boundaries into discrete mesh elements. To obtain a 3D volumetric mesh model, mesh generation can be performed. Mesh generation is the practice of generating a mesh, which is the subdivision of a continuous geometric space into discrete geometric and topological cells. Mesh generation is generally performed with linear (affine) mesh elements. The use of linear mesh elements may not adequately represent the boundaries of curved domains in the 3D model geometry unless a very fine mesh is used. An overly fine mesh can lead to undesirably high computational costs and, in some cases, prevent converged simulations using a fine mesh. Utilizing curvilinear mesh elements can enable the use of large mesh elements with a high base order while still capturing important geometric details.

[0101] In particular, curved meshes have been shown to be efficient for simulating sound propagation using wave-based simulations. Curved sides and edges provide high-fidelity results. As disclosed herein, this becomes particularly effective, along with the possibility of using wave-based solvers on larger meshes that apply finite element methods (FEMs), such as the discontinuous Galerkin finite element method (DGFEM), higher-order discontinuous Galerkin finite element method, or the spectral element method (SEM). In particular, DGFEM and SEM have been shown to be particularly effective finite element methods for handling curved mesh elements.

[0102] A curvilinear mesh may be defined by curvilinear coordinates, in contrast to a rectilinear mesh, which is defined using affine coordinates. Curvilinear coordinates are a coordinate system in Euclidean space in which coordinate lines may be curved. These coordinates may be derived from a set of Cartesian coordinates by using a transformation that is locally invertible at each point. Well-known examples of curvilinear coordinate systems in three-dimensional Euclidean space are cylindrical coordinates and spherical coordinates.

[0103] In one embodiment, the air volume is represented as a volumetric 3D mesh model. Providing the volumetric 3D mesh can be done, for example, by discretizing the air volume into a mesh of cells or a grid of points. The air volume can be defined by a previously defined 3D model of the virtual space. The air volume can be any fluid or environment that can potentially propagate acoustic waves.

[0104] A qualitative assessment is performed on the volumetric 3D mesh model to evaluate critical areas of sound propagation. The qualitative assessment may be performed, for example, by assessing the uniformity of the volumetric 3D mesh model. Having a uniform volumetric 3D mesh promotes consistent and efficient processing, for example, when processing wave-based simulations on parallel GPUs or other types of processing units. The time step here is the same for each element, a direct result of the fact that the smallest element in the mesh globally constrains the time step.

[0105] Therefore, ideally, all elements should have similar characteristic lengths (e.g., within 10%) to avoid unnecessarily increasing the computation time of the simulation.

[0106] The volumetric mesh may be, for example, a tetrahedron, which has been shown to be an efficient geometry for capturing desired geometric features and boundary surfaces of a 3D model. By comparing each tetrahedron mesh element to a standard tetrahedron mesh element, a uniformity score can be established for each volumetric tetrahedron mesh and / or for the entire volumetric 3D mesh model.

[0107] The uniformity score can be used, for example, to indicate how fast the simulation will take. If the uniformity is high, the simulation is as efficient as possible, and the frequency range of the wave-based solver simulation, e.g., the first acoustic frequency range, can be expanded, for example. If the uniformity is low, the frequency range of the wave-based solver simulation, e.g., the first acoustic frequency range, can be reduced, for example. Alternatively, the volumetric 3D mesh model can be remeshed, or the entire 3D model of the virtual space can be redesigned to improve uniformity.

[0108] Once the virtual space has been meshed or discretized in some way, the acoustic wave propagation problem can be solved using an acoustic wave equation module. Generally, wave-based methods can divide the virtual space into subdomains to solve algebraic equations in each subdomain instead of solving the entire virtual space at once. The discretization of the virtual space can depend on the wave-based method used to solve the equations. Wave-based methods such as the finite-difference time-domain (FDTD) method, the finite element method (FEM), or the boundary element method (BEM) can be used.

[0109] A volumetric 3D mesh model of an air volume can represent such a discretized subdomain. Thus, each volumetric 3D mesh, e.g., a tetrahedron, provides a discretized subdomain. Acoustic waves can then be expressed, for example, in terms of polynomial basis functions on the subdomain. The discretized subdomain can represent a reference element by which differential equations characterizing acoustic wave propagation can be discretized into algebraic equations.

[0110] Thus, a 3D model, e.g., a virtual domain, can be subdivided into smaller domains. The continuous partial differential equations can then be converted into discrete algebraic equations that can be solved by a processing unit. Thus, discretization can generally be thought of as a two-step process: the first step is to discretize the virtual domain, and the second step is to solve the numerical equations for each element of the discretized virtual domain.

[0111] A discontinuous Galerkin finite element method is applied to each subdomain to solve the partial differential equations that simulate sound propagation in each separate subdomain. Using the DGFEM for this application exploits the discontinuous nature of the Galerkin method, allowing sound propagation in each subdomain to be simulated independently, without relying on neighboring subdomains. This provides a method and solver that is highly suitable for parallel processing on heterogeneous multi-device, many-core computing architectures, e.g., on the cloud, since simulations can be run independently for each subdomain and then combined when the simulations are complete. The discontinuous Galerkin method is also attractive because of its ability to work with unstructured meshes, making it easier to handle complex geometries.

[0112] The discontinuous Galerkin method may have the advantage of efficient parallelization and the ability to handle complex geometries in 2D or 3D virtual domains. Other methods, such as the FDTD method, are efficient in parallel processing but cannot handle complex geometries. FEM, BEM, and FVM can handle complex shapes but may not perform efficiently on parallel computing.

[0113] The DG module may include a locally responsive boundary model submodule for simulating sound propagation in a boundary region including a first material. The locally responsive model may refer to the locally responsive behavior of the boundary region. When a boundary is locally responsive, this means that the reflective properties of any point on the boundary region are a completely local phenomenon and can be completely characterized by a locally responsive surface impedance. The boundary may be frequency dependent, meaning that the reflected wave has both a different phase and amplitude than the incident wave, and such variation varies with frequency. The boundary impedance may be replaced by a frequency-dependent complex-valued impedance to account for this effect.

[0114] For example, for locally reactive materials, material modeling involves the internal solution.

number

number

number

number

number

number

[0115] Waves reflected from an acoustic boundary at a given location

number

number

number

number

number

number

[0116] Characterizing material properties in terms of complex reflection coefficients, as used here, directly relates to characteristic waves, making it a natural choice for flux-based solvers. This method also has crucial advantages over the more common impedance or admittance formulations of surface pressure and velocity for iterative time-domain solvers, such as (i) numerical stability can be easily ensured a priori, as discussed herein, and (ii) it allows modeling of materials with very high and / or very low impedances. Boundary conditions based on impedance or admittance certainly have some inherent limitations in this context, since these quantities essentially characterize material properties in isolation and do not take into account the surrounding air. In mathematical terms, pressure and velocity are not characteristically independent, so impedance or admittance cannot be considered as a general, linear, time-invariant system that describes the entire reflection process [5].

[0117] The local response assumption for characterizing the absorption properties of an interface surface may generally be considered acceptable for rigid or near-rigid surfaces, as well as porous materials with high fluid resistivity attached to a rigid backing. This type of material may define what we will call the first material, as discussed herein.

[0118] The aforementioned local boundary modeling approach involves modeling the acoustic properties of a material from its surface properties. This is generally a good physical approximation when the material can be considered to react locally, which assumes that the wave propagation is inside the material in a direction normal to the surface. This approach is also beneficial from a numerical point of view, as it does not require meshing the volume of the absorbing material, saving computational time.

[0119] However, in some situations, for example, in thin porous layers lined by air cavities, the material properties may depend on the angle of incidence of the acoustic wave, and local reactive neighbors may introduce errors. Such bulk reactive (or extended reactive) materials can be accounted for by explicitly solving specific propagation equations inside the material, for example, via equivalent fluid approaches [3, 6, 7] or fluid-structure interaction simulations.

[0120] The DG module can also include an extended reactive boundary model submodule to simulate sound propagation in a boundary region that includes a second material. The extended reactive boundary model can take into account the surface impedance of the boundary region, which can vary with the angle of incidence of the incoming sound wave, for non-local sound propagation along the boundary surface. By modeling sound propagation within a boundary material, such as a porous material or structure, and coupling this with an air-domain simulation, extended reactive behavior can be modeled.

[0121] In one embodiment, the extended response frequency boundary model submodule is configured to model a second material in a fluid-structure acoustics simulation, which may be similar to a fluid-porous simulation, which may be useful for modeling vibrating panels such as lightweight gypsum walls.

[0122] The extended response frequency sub-module for simulating sound propagation in boundary regions may be relevant for surfaces with elastic properties or fluid layers, such as soft ground, porous materials lined with air cavities, airtight membranes, or perforated panels. These types of surfaces may have acoustic waves propagating along the surface, and therefore the extended response frequency sub-module may be relevant to obtain accurate behavior of the acoustic wave propagation.

[0123] In one embodiment, the 3D model includes a directional sound source for emitting sound in a prescribed direction. The directional sound source can be modeled as a directional sound source. The prescribed direction may be a specific angle and / or a specific orientation of the directional sound source. The directional sound source can be achieved in several ways. One method may include building a box with a movable piston in virtual space to emulate a speaker. Another method may model radiation from a directional source using, for example, a mathematical model based on spherical harmonics, where the governing differential equations are different (forced equations vs. unforced in the normal model of propagation).

[0124] In a preferred embodiment, the 3D model includes a directional source for emitting sound in a prescribed direction, which is modeled using a broadband point-source excitation, which may be generated, for example, via a forcing term in the governing equations or via an initial pressure disturbance.

[0125] One possible way to model a directional source in a 3D model is to apply impulse surface velocity excitation to a speaker box membrane, preferably to include the speaker box geometry. The speaker box membrane may be a diaphragm, which may be a transducer intended to convert mechanical vibrations into sound or vice versa. The speaker box membrane may generally be a sheet of various materials suspended by its edges. By applying impulse surface velocity excitation to the speaker box membrane, a directional sound source can be obtained due to the membrane's radiation pattern and subsequent edge diffraction that the speaker box geometry may introduce.

[0126] A speaker box may include at least one driver. In the context of a speaker box, a driver may be a discrete transducer capable of converting an electrical audio signal into sound waves. This may also be referred to as an electrodynamic speaker driver or speaker driver. The driver may include a diaphragm, as defined herein, which may move back and forth to generate pressure waves. The driver may be attached to a rigid enclosure, preferably made of wood, plastic, or possibly metal. The rigid enclosure may preferably be defined as a speaker enclosure or speaker box. The rigid enclosure may isolate acoustic energy from the front of the diaphragm from acoustic energy behind the diaphragm. Preferably, a horn may be used to increase the efficiency and directionality of the directional sound source. The horn may be a horn loudspeaker, consisting of an acoustic horn that can increase the overall efficiency of the driving element. A common form may consist of a flared duct that conducts acoustic or sound waves into a virtual room or the real-world outside air. As described herein, horns can be used to improve the coupling efficiency between a speaker driver and a propagation medium for acoustic waves.

[0127] The loudspeaker box may include at least two membranes, a first membrane having a substantially larger diameter than a second membrane, the first membrane being capable of modeling a directional sound source in a first directional source frequency range, and the second membrane being capable of modeling a directional sound source in a second directional source frequency range. The first directional source frequency range may be comprised between 20 Hz and 1 kHz, preferably between 20 Hz and 2 kHz, and more preferably between 20 Hz and 3 kHz. The second directional source frequency range may be comprised between the upper limit of the first directional source frequency range and 4 kHz, preferably 5 kHz, more preferably 6 kHz, even more preferably 7 kHz, most preferably 8 kHz, and even most preferably 10 kHz. By including at least two membranes in the loudspeaker box, a better simulation of a directional sound source can be performed, since the loudspeaker box preferably includes multiple membranes, a first membrane preferably designed for the low frequencies of the audio frequency spectrum and a second membrane preferably designed for the high frequencies of the audio frequency spectrum.

[0128] The first membrane may be a woofer and the second membrane may be a tweeter, which are common types of drivers or membranes in a loudspeaker or speaker box.

[0129] In another embodiment, the loudspeaker is excited by applying a normal acoustic velocity along the surface of the membrane geometry. The membrane geometry may be fixed. The normal acoustic velocity along the surface of the membrane geometry may be specified by a Neumann boundary condition. Due to numerical constraints, the excitation signal may be a Gaussian-like impulse. Such excitation may have a substantially flat energy spectrum at all frequencies within the audible range, i.e., 20 Hz to 20 kHz. Advantageously, a post-processing step can normalize the simulated room impulse response to a simulated free-field on-axis response at a distance of 1 meter.

[0130] Preferably, the rigid enclosure of the loudspeaker box can be modelled as an acoustically hard boundary. The rigid enclosure of the loudspeaker box can preferably be arranged substantially around the at least two membranes.

[0131] The approach described in the previous paragraph allows for a simple implementation of directional sound sources, which may preferably require a sufficiently accurate model of the speaker box geometry. The effects of diffraction from the baffle can be inherently included in the simulation, making it particularly suitable for low-frequency simulations. A baffle may refer to a flat or panel-like structure arranged in front of at least one speaker driver. One function of the baffle may be to provide a mounting surface for at least one speaker driver and to separate the front and rear sound waves generated by the driver. By separating the front and rear sound waves, the baffle may help prevent them from canceling out.

[0132] A wave-based solver may be used to simulate acoustic wave propagation at a point in time. However, if it is desired to simulate the time propagation of waves, it is necessary to simulate time-dependent wave propagation. Therefore, in one embodiment, the DG module may include a time-marching submodule that includes at least a first time-marching method for simulating the propagation of sound in time and space. The first time-marching method may be an explicit time-step method. To solve the acoustic wave propagation equation, it may be necessary to perform a time discretization. The time discretization of a semi-discrete numerical system may include the integration of every term in the different equations over a predefined time step.

[0133] A semi-discrete system may be a system of partial differential equations (PDEs) where the spatial part can be discretized but the time part cannot. A semi-discrete system may also be a system of ordinary differential equations, i.e. the PDEs are transformed into ordinary differential equations.

[0134] The time-marching method may include a second time-marching algorithm to simulate the propagation of sound in time and space. Using a second time-marching algorithm may be relevant in certain conditions, for example, when the boundary equations impose strict stability criteria that make the first time-marching algorithm inefficient.

[0135] For example, the first time-marching algorithm may be a commonly used time-marching algorithm, such as the described low-capacity explicit Runge-Kutta algorithm. These are typically fast and effective. However, they are not necessarily very robust unless very small time steps are used; that is, if the simulation is too complex, the time-marching algorithm may fail and return errors and erroneous results.

[0136] Therefore, when the simulation is too complex, a second, more robust time-marching algorithm may be used.

[0137] The wave-based solver further includes criteria for establishing when the simulation becomes too complex and therefore applies the second time-marching algorithm.

[0138] In one example, the time march can be further considered with a spatial discretization defined for a given time t in [Equation (9)], from which the semi-discrete equation (discrete in space, continuous in time) is derived:

number

number

number

number

number

number

[0139] When time marching, an appropriate time step size must be chosen (see details below on numerical stability). The main wave equation and ordinary differential equations associated with boundary conditions may impose constraints on the time step.

[0140] As discussed in one embodiment, two time-marching algorithms can be used. The default time-marching algorithm can be, for example, the standard low-volume explicit Runge-Kutta (LSERK) algorithm, which requires only two storage locations per variable. This is an efficient time-marching algorithm. Fourth-order accuracy is typically used in the time integration, which, thanks to the linearity of the wave equation [10, 11, 12], results in a four-stage scheme. This time-marching algorithm is selected when the time step size conditions of the wave equation are stricter than those of the boundary equation. However, if the boundary equation imposes stricter conditions, a so-called implicit-explicit time-marching algorithm may be adopted [1].

[0141] In a preferred embodiment, the wave-based solver includes a unique time-marching algorithm, which is a simplified Runge-Kutta algorithm, which is used in conjunction with simplified integration. By using only the simplified Runge-Kutta algorithm, the simulation is much faster than using the first and second time-marching algorithms, as discussed herein.

[0142] When time-stepping, the numerical stability of the algorithm must be considered. For example, in some embodiments, the time integration is performed using a semi-discrete operator over each element:

number

number

number

number

number

number

number

number

number

[0143] In one embodiment, the presence of absorbing materials in the simulation can affect the modulation stability because it increases the stiffness of the spatial operator. In the case of locally reactive materials modeled with their reflection coefficients, all polar regions associated with the boundary material

number

number

number

[14] . In such cases, local explicit time stepping can also be used.

[0144] The DGFEM method has been shown to be amenable to parallel computing, but it has also been shown that parallelization can be significantly impacted if not properly implemented.

[0145] In one embodiment, a Message Passing Interface (MPI) is used to handle communication between central processing units (CPUs), which facilitates multi-device compatibility, i.e., the cooperation of several computers (CPUs) to solve a given problem. MPI works by treating each CPU core as a unique entity with its own separate memory. Information can then be shared between CPU cores via a messaging interface.

[0146] In one embodiment, a virtual domain (e.g., virtual space) is divided into chunks. For example, the virtual domain is divided or discretized into a set of non-overlapping, potentially unstructured mesh elements or subdomains. Each chunk may also be a set of mesh elements. Dividing the domain into equal chunks is preferable to balance the load among CPU cores. However, it may be difficult to determine in advance the computational load associated with each element. If a significant computational load imbalance exists among domain chunks, some CPUs / graphical computer units (GPUs) may become idle at every time step, severely adversely affecting performance. Load balancing can be ensured by estimating in advance the amount of computational operations required for each element on the mesh and then ensuring that each chunk requires a similar number of computational operations. Each CPU core stores simulation information required for calculations on elements / chunks belonging to its portion of the domain. However, some CPUs share interfaces between them, where information must be shared for acoustic waves to propagate from a domain chunk processed by one CPU to an adjacent domain chunk processed by another CPU. In other words, the numerical flux is calculated based on neighboring elements located on different CPUs, which may require, for example, that MPI communication between cores take place in a so-called halo exchange.

[0147] The code is structured in one embodiment to assume that each CPU core has access to one GPU. This creates interface communication between CPU cores compared to using only the CPUs, which can appear to increase costs, such as bandwidth costs, because CPU0 communicates this information to the CPUs via MPI. x This is because CPU0 may copy all interface information from GPU0 to RAM on CPU0 before it can communicate with GPU0.

number

number

[0148] In one example, an implementation of the above LSERK time-march method, including all interface communications, is summarized in Algorithm 1. A GPU function, i.e., a kernel, is launched for each iteration of the time-stepping algorithm. [Table 1] In modern graphics cards, memory movement between pinned host memory and device memory can be performed using an on-board direct memory access (DMA) engine, which can fully overlap with computational activity on the GPU die. This allows device-to-host communication and the volume integral calculation from equation (4) above to begin simultaneously, as shown in step 1 of Algorithm 1. Communication is preferably performed in steps 1, 2, and 3 before the volume kernel finishes. This reduces communication costs, which is possible because the volume kernel is completely asynchronous to the host. Step 4 cannot begin before all communication in steps 1, 2, and 3 has finished and the volume kernel has finished.

[0149] Each subdomain is solved independently on a single CPU core. A single CPU core may be defined as a single unit that can receive instructions and perform calculations or operations to fulfill those instructions. A CPU may have multiple cores.

[0150] In one embodiment, the wave-based solver further includes extracting one or more wave impulse response(s) based on a simulation of sound propagation. The one or more wave impulse response(s) can be extracted based on a simulation performed in a virtual domain as described herein, where the simulation model is excited with an impulsive excitation, i.e., a short-duration, broadband signal.

[0151] The wave impulse response can be a spatial impulse response. A spatial impulse response embeds spatiotemporal information about the direction of arrival of an incoming acoustic wave at a receiver location. Typically, a spatial impulse response includes multiple single-channel impulse responses, each recording sound from a specific direction or angle at the same listening point. A spatial impulse response can be constructed in a simulation by recording the impulse response of a room at multiple locations around the receiver location using a virtual microphone array, e.g., a spherical microphone array. The receiver can be either omnidirectional or cardioid, and the microphone array can be either open-ended or rigid. The microphones in the array can be distributed in many ways. The number of microphones used in the virtual microphone array can be selected based on the degree of spatial resolution required in the resulting spatial impulse response. Once the impulse responses are simulated with all the virtual microphones, spherical harmonic techniques, such as Ambisonics, can be used to encode the spatial impulse response.

[0152] The spatial characteristics around the receiver location can be captured to enable binaural or multi-channel loudspeaker auralization. An array of receivers can be arranged around the receiver location to determine the direction of the sound waves. This information can be used, for example, to encode the resulting sound field into an Ambisonics spatial impulse response. The spatial impulse response can be decoded into a binaural or multi-channel loudspeaker playback system.

[0153] In one embodiment, a spherical receiver array is arranged around at least one receiver. The spherical receiver array may be an open spherical array of cardioid receivers. This array design is robust and allows for a wide operating frequency range.

[0154] The spherical receiver array may include at least two receivers, preferably at least four receivers, more preferably at least eight receivers, even more preferably at least 16 receivers, most preferably at least 32 receivers, and even most preferably at least 64 receivers. The more receivers arranged in the spherical receiver array, the higher order Ambisonics can be obtained.

[0155] The number of receivers in the array may determine the maximum truncation order N that can be considered for the spherical harmonic decomposition. The sound field may be sampled with at least as many receivers as there are terms used in the expansion,

number

[0156] The wave-based solver may include an output module that can render the DG simulation. The output module can provide various formats of output from the simulation, such as visual, graphical, numerical, and / or audible output. Sound rendering uses simulated sound propagation to convey emotions or effects related to a virtual situation. Rendering can convey emotions associated with a sound source or the acoustics of a virtual domain. Sound rendering is an essential element of auralization. The complexity of sound rendering methods increases dramatically in virtual domains with one or more dynamic receiver(s) or source(s). The disclosed wave-based solver and computer-implemented methods related to wave-based solvers can address the computational complexity caused by dynamic receiver(s) and / or source(s). In some cases, sound may arrive at a receiver from many previous source locations. As a result, multiple impulse responses must be calculated between many previous source locations and the current receiver location.

[0157] The graphical output may include visualization of at least a portion of the DG simulation in a cut plane of a 3D model of the virtual space. The portion of the DG simulation is a graphical representation of the energy levels of acoustic waves spatially represented in the virtual domain in which the simulation was performed. The graphical output allows a representation of the energy levels to evolve within the virtual domain in time, allowing a user to visualize the evolution of the energy levels of acoustic waves within the virtual domain at a precise time and / or location within the virtual domain. The graphical output may also be configured to provide visualization of acoustic waves within the virtual 2D domain with the same functionality as described herein. The graphical output may also be a static spatial color map of various sound descriptors, such as sound pressure level, reverberation time, intelligibility, or other acoustic parameters.

[0158] The numerical output may include one or more possible formats or forms, such as impulse response, frequency response, energy decay curve, and / or acoustic parameters such as reverberation time, intelligibility, sound pressure level and / or speech intelligibility. More generally, the numerical output may include parameters such as those described in

[16] .

[0159] The audible output includes playing an impulse response convolved with a sound file. A sound file may be defined as a sound, such as speech or music, processed by a processing unit for use by a processing unit or a second processing unit. The convolution of the sound file with the impulse response may render the sound file in a virtual reality setting. In the case of spatial (multi-channel) impulse responses, spatial decoding techniques such as Ambisonics decoding may be applied to render the multi-channel output into a binaural auralization, for example, for headphone playback, or into a multi-channel auralization, for example, for a loudspeaker system.

[0160] As discussed, it is generally known to use geometric acoustic solvers to simulate wave propagation. However, there is room for improvement in known geometric acoustic solvers, particularly to optimize geometric solvers for use in conjunction with the wave-based solvers described herein.

[0161] In one embodiment of the geometric acoustic solver, at least a first acoustic ray tracing module determines at least a first ray tracing simulation of a first portion of the sound propagation, and at least one image source module determines at least one image source simulation of a second portion of the sound propagation.

[0162] In particular, an image source module can be used to determine the first reflections, e.g., first, second, and third order reflections. Compared to ray tracing, the image source module generates a more accurate model of specular reflections used to estimate wave propagation, but is computationally more intensive, especially as the number of reflection orders increases. Therefore, a ray tracing module is used to determine the remaining number of reflections, because it is computationally faster, but not more accurate.

[0163] It should be understood that the image source module may be used to determine fewer or more than the first three reflections, and can be determined by the user or application to achieve optimal accuracy and / or processing time in a particular case. The geometric acoustic solver can also provide estimates regarding the accuracy and / or processing time of the simulation before the simulation is started by the user.

[0164] In further embodiments, switching between the image source module and the ray tracing module may be determined by the energy of the ray rather than the number of reflections. Thus, the geometric acoustic solver may switch from the image source module to the ray tracing module once the energy of the simulated sound propagation falls below a certain threshold set by the user or software for a particular application, or to achieve a certain accuracy and / or processing time.

[0165] In one embodiment, the first part of the sound propagation is the initial set of reflections of the sound propagation, and the second part of the sound propagation is the subsequent reverberations of the sound propagation.

[0166] For example, the initial set of reflections may be the first, second and third reflections, and subsequent reflections may be the fourth, fifth and subsequent reflections.

[0167] In another embodiment, the geometric acoustic solver further includes a second acoustic ray tracing module for determining a second ray tracing simulation by simulating sound propagation, and the hybridization module further combines the second ray tracing simulation with the first ray tracing simulation and combines at least one image source simulation into the geometric acoustic simulation.

[0168] In one embodiment, the first acoustic ray tracing module determines a first sub-part of the first portion of the sound propagation, and the second acoustic ray tracing module determines a second sub-part of the first portion of the sound propagation.

[0169] In one embodiment, the geometric acoustic solver is further modified for high-performance computing implementation by parallelizing the computational operations performed by the solver, for example, by parallelizing the propagation of each ray, parallelizing different frequency ranges (e.g., different octave bands) simulated by the geometric acoustic solver or by each image source branch in the image source tree, which can be implemented on a multi-core CPU computing architecture or a GPU computing architecture.

[0170] As mentioned above, it is common to derive an impulse response from the acoustic simulation, which is then used to render acoustic sounds at specific points within a virtual room.

[0171] Thus, in one embodiment, the geometric acoustic solver may further include a geometric impulse response module for extracting a geometric impulse response based on a geometric acoustic simulation of sound propagation.

[0172] A geometric impulse response can be, for example, mono or spatial. A mono impulse response records all sounds at one point without considering the direction of the sound. A spatial impulse response records the direction from which the sound comes. Typically, a spatial impulse response contains multiple mono impulse responses, each recording sounds from a specific direction or angle at the same point, for example using spatial harmonic encoding techniques.

[0173] The spatial characteristics around the receiver location can be captured to enable binaural or multi-channel loudspeaker auralization. An array of receivers can be arranged around the receiver location to encode the resulting sound field into an Ambisonics spatial impulse response. The spatial impulse response can be decoded into a binaural or multi-channel loudspeaker playback system.

[0174] In one embodiment, a spherical receiver array is arranged around at least one receiver. The spherical receiver array may be an open spherical array of cardioid receivers. This array design is robust and allows for a wide operating frequency range.

[0175] The spherical receiver array may include at least two receivers, preferably at least four receivers, more preferably at least eight receivers, even more preferably at least 16 receivers, most preferably at least 32 receivers, and even most preferably at least 64 receivers. The more receivers arranged in the spherical receiver array, the higher order Ambisonics can be obtained.

[0176] The number of receivers in the array may determine the maximum truncation order N that can be considered for the spherical harmonic decomposition. The sound field may be sampled with at least as many receivers as there are terms used in the expansion,

number

[0177] The geometric acoustic solver may include an output module capable of rendering the geometric acoustic solver simulation. The output module may provide various formats of output from the simulation, such as visual, graphical, numerical, and / or audible output. Sound rendering uses simulated sound propagation to convey emotions or effects related to a virtual situation. Rendering can convey a feel associated with a sound source or the acoustics of a virtual domain. Sound rendering is an essential element of auralization. The complexity of sound rendering methods increases dramatically in virtual domains with one or more dynamic receiver(s) or source(s). The disclosed wave-based solver and computer-implemented methods related to wave-based solvers may address the computational complexity caused by dynamic receiver(s) and / or source(s). In some cases, sound may arrive at a receiver from many previous source locations. As a result, multiple impulse responses must be calculated between many previous source locations and the current receiver location.

[0178] In a geometric acoustics solver, the graphical output includes visualizing at least a portion of the geometric acoustics simulation, for example, by visualizing rays traced over a 3D domain over time. Another graphical output consists of visualizing effective image sources representing effective specular reflection paths in the simulation. The graphical output can also be static spatial color maps of various sound descriptors, such as sound pressure level, reverberation time, intelligibility, or other acoustic parameters.

[0179] In one embodiment, the geometric acoustic solver relies on the impedance of the boundaries of the air volume of the virtual domain. The geometric acoustic solver can be configured to use the boundary impedance to define boundary conditions, thereby accounting for both energy reduction and phase shift at the boundaries of the virtual domain. This can be implemented, for example, by tracking the phase shift introduced at the boundaries of all reflections of a ray or all generated image sources, and accumulating the total phase shift that occurs as the acoustic wave travels between the source and the receiver. Furthermore, it can mean that the phase of the wave needs to be tracked for each ray extent.

[0180] Furthermore, the merging or hybridization step involves the time arrival, i.e., phase, of the propagated sounds in the virtual space arriving at the receiver location simultaneously. In one embodiment, the geometric acoustics-based solver is configured to use a pressure-based image source method, where the pressure-based image source method models the phase shift at the boundary conditions.

[0181] In a preferred embodiment, the virtual domain includes at least one directional sound source for emitting sound in a prescribed direction. The geometric acoustic solver is capable of handling directional sound sources. The geometric acoustic solver is configured to model directional sound sources and can simulate sound propagation when one or more sound sources are directional.

[0182] In one embodiment, a geometric acoustic solver approximates the diffraction of acoustic waves.

[0183] As discussed herein, computer-implemented methods for generating impulse responses for listening points within a room can be used in architecture. The simulated and generated impulse responses and related auralization techniques can be extremely useful in making timely and informed decisions during the architectural design process and modifying potential designs to avoid undesirable acoustic effects in the design. Different indoor acoustic conditions can be experienced without physically realizing them during the design and construction phase, also known as virtual prototyping, until a client and / or building owner finds one they are satisfied with. All parties involved in the design and construction process can understand the value of good acoustic conditions by listening to the simulated sound, for example, by listening to rendered audio sounds at listening points within a 3D model of the room in which the acoustics may be simulated.

[0184] Similarly, the impulse responses generated by the computer-implemented methods discussed herein can benefit the audio industry, which can test how audio devices sound in realistic room conditions to optimize their designs with respect to directional patterns, radiation characteristics, speech separation performance, background noise rejection, direction of arrival estimation, etc. In this way, audio devices can adapt their sound to maximize their performance for given acoustic conditions characterized by the impulse response(s) generated by the computer-implemented methods as discussed herein. Audio devices can include smartphones, smart speakers, televisions, traditional loudspeakers, hearing aids, audio devices attached to head-mounted displays, and any combination thereof.

[0185] By generating impulse responses with the computer-implemented methods described herein, metaverse and entertainment applications related to the entertainment industry, including movies and games, can recreate high-fidelity sound propagation within virtual environments. Any unexplored acoustics can be freely defined and simulated in a virtual scene, allowing users to experience immersion while playing games, watching movies, or any combination thereof. Simulation of acoustics by generating impulse responses with computer-implemented methods can be a promising tool in automating realistic audio signals for a given scene without significant human interaction.

[0186] In general, the acoustic impulse response of a system represents the system's characteristics and can therefore be used to evaluate room designs from a specific listening point. Thus, the frequency content of the impulse response can indicate whether certain frequencies are dominant and, if undesirable, whether they should be eliminated. The system can then be designed or the room redesigned to change the room's acoustic characteristics. Thus, impulse responses provide a powerful technical tool for designers, architects, and acoustic engineers to evaluate the acoustic characteristics of spaces such as rooms.

[0187] As discussed, one aspect of applying and using acoustic simulation as discussed herein can be the training of machine learning driven audio algorithms.

[0188] The machine learning-driven audio algorithms described herein can be used to improve signal processing of acoustic waves for machine learning-driven audio devices. When processing sound propagation, audio devices have difficulty distinguishing and separating key information in transmitted and / or received sound. Sound propagation may be affected by diffraction, reverberation, or echo, which may cause noise or undesirable effects in the sound that may contain the key parts of the transmitted and / or received information. By training a machine learning-driven audio device with several machine learning-driven audio algorithms, the audio device can automatically and independently address sound propagation deficiencies in a specific area. The specific area may be any area such as an office, an outdoor environment, a semi-enclosed room, a means of transportation such as a car or bus, a house or building, etc., so training the machine learning-driven audio algorithm in as many different environments as possible may be necessary to improve sound quality by distinguishing between key information contained in the sound and undesirable effects and / or sounds that may need to be removed.

[0189] A standard approach used consists of generating data by measuring the propagation of sound in real-world scenarios, such as different rooms with different reverberations, with a sound source and one or more microphone(s) arranged in the different rooms. This allows data to be generated that can then be used to train machine learning-driven audio algorithms. These methods are well known, but are expensive and time-consuming.

[0190] As disclosed herein, the ability to generate simulated synthetic data for machine learning-driven audio algorithms in a fast and accurate manner can alleviate expensive and time-consuming methods of performing measurements to gather data, potentially making machine learning-driven audio algorithms more accessible, accurate, and / or cost-effective, as these algorithms may rely on amounts of synthetic data that cannot be generated using current methods available on the market.

[0191] The at least one 3D model may include at least one interior surface, such as a surface of at least one virtual object within the 3D model. The at least one virtual object may be furniture, a plant, a car, or a frame. The at least one interior surface may be any surface contained in or within the 3D model. By object, it may be understood as something static. A static person may also be included within the at least one 3D model. A surface may be defined as the exterior or top layer of something.

[0192] The at least one sound receiver may include a plurality of oriented receivers. The plurality of oriented receivers may be arranged at the location of the at least one sound receiver, where a first oriented receiver may be oriented at a first angle and a second oriented receiver may be oriented at a second angle. The plurality of oriented receivers may be oriented to capture different impulse responses at the location of the at least one sound receiver. This may be useful for obtaining spatial impulse response outputs from the simulation.

[0193] The spatial characteristics around the receiver location can be captured to enable binaural or multi-channel loudspeaker auralization. An array of receivers can be arranged around the receiver location to encode the resulting sound field into an Ambisonics spatial impulse response. The spatial impulse response can be decoded into a binaural or multi-channel loudspeaker playback system.

[0194] In one embodiment, a spherical receiver array is arranged around at least one receiver. The spherical receiver array may be an open spherical array of cardioid receivers. This array design is robust and allows for a wide operating frequency range.

[0195] The spherical receiver array may include at least two receivers, preferably at least four receivers, more preferably at least eight receivers, even more preferably at least 16 receivers, most preferably at least 32 receivers, and even most preferably at least 64 receivers. The more receivers arranged in the spherical receiver array, the higher order Ambisonics can be obtained.

[0196] The number of receivers in the array may determine the maximum truncation order N that can be considered for the spherical harmonic decomposition. The sound field may be sampled with at least as many receivers as there are terms used in the expansion,

number

[0197] The at least one sound source may be at least one directional sound source. The at least one directional sound source may emit sound in a specific direction. The at least one directional sound source may be modeled in many different ways. One method of modeling the at least one directional sound source may include building a 3D model of a speaker around the at least one sound source, where the speaker guides the sound emitted from the at least one sound source in a specific direction within the 3D model. The speaker may be constructed of at least one material, which may have properties that absorb or reflect sound waves so that a majority of the sound waves are directed or guided in a specific direction within the 3D model of the virtual space. The source may also be omnidirectional.

[0198] The at least one audio source may be configured as described herein.

[0199] The method may further include acquiring at least one raw sound data. The at least one raw sound data may be an audible sound, such as music and / or a human voice. The at least one raw sound data may not include any echo and / or reverberation, such as sounds recorded in an anechoic chamber. The at least one raw sound data may be acquired by recording it in an anechoic chamber. The at least one raw sound data may include frequencies between 20 Hz and 20 kHz, which may be defined as audible frequencies. The frequency spectrum may extend slightly beyond these ranges.

[0200] In acoustic simulation of a virtual space, the impulse response of a given virtual domain can be simulated using a wave-based solver, or a geometric acoustic solver, or a hybrid of a wave-based solver and geometric acoustics, as described herein. The impulse response can be mono or spatial (multi-channel). At least one raw sound data can then be convolved with the impulse response to generate modified sound data, a simulation of what this raw sound would actually be like in this virtual space. By knowing the raw sound (input) and the modified raw sound (output), machine learning algorithms can be trained to understand the relationship between the raw sound and the modified raw sound, allowing the machine learning algorithm to perform various acoustic processing such as dereverberation.

[0201] Training data for a machine learning model or algorithm is data used to train the machine learning model or algorithm to accurately predict the particular outcome or answer that you want the model to predict.

[0202] The machine learning model can be trained by comparing at least one raw sound data, such as a system input, with at least one modified raw sound data, such as a system output. The comparison between the at least one raw sound data and the at least one modified raw sound data teaches the machine learning model to recognize differences between the at least one raw sound data and the at least one modified raw sound data, enabling the machine learning model to remove unwanted or undesirable effects, such as echo and / or reverberation, in the at least one modified raw sound data.

[0203] The machine learning model may also be trained by convolving at least one raw sound data with a simulated impulse response obtained by at least one sound propagation simulation.

[0204] The input data may include multiple three-dimensional models of the virtual space, preferably three or more three-dimensional models of the virtual space, more preferably ten or more three-dimensional models of the virtual space. The quality, quantity, and diversity of the input data determine the accuracy and performance of the machine learning model or algorithm. The more input data the model can be trained with, the better the machine learning model's performance and the wider the range of situations the model can handle. Therefore, it may be necessary to generate as much input data as possible by building as many three-dimensional models of the virtual space as possible and run sound propagation simulations on each of these 3D models.

[0205] These 3D models can be automatically generated to create a relatively large number of 3D models in a short period of time, potentially making the methods described herein more rapidly generate large amounts of useful synthetic data for training machine learning-driven audio algorithms.

[0206] In some specific applications, at least one sound source may be substantially co-located with at least one sound receiver. Indeed, this configuration may be necessary to efficiently train machine learning-driven audio devices, such as audio devices that perform blind room response modeling. In some cases, loudspeakers, sound bars, televisions, or smart speakers may attempt to adapt or filter the sounds they emit in the room or area in which they are installed to improve sound quality for potential listeners. To adapt or filter the sounds they emit in the room or area in which they are installed, they typically emit at least one sound signal and record at least one return sound signal that returns to the speaker with a microphone. The speaker then estimates room characteristics from the at least one return sound signal using a machine learning model or algorithm, and then adapts or filters the sounds it emits to limit undesirable or unwanted sound effects.

[0207] The at least one sound propagation simulation may have at least one simulation time, which may be a compromise between accuracy of the simulation and the time spent by the computer to run and complete the at least one sound propagation simulation.

[0208] At least one simulation time may be calibrated according to properties of the 3D model(s).

[0209] The properties of the 3D model(s) may include the geometry and / or size of the 3D model(s).

[0210] The method can be configured for use in speech recognition, echo cancellation, beamforming, direction of arrival estimation, blind source separation, blind room response modeling, or feature extraction. The configuration of the method depends on various applications, such as those listed as examples. By configuration, this may mean that the steps of the method can be performed in a different order, resulting in optimized training of machine learning-driven audio algorithms.

[0211] In one embodiment, at least one sound propagation simulation is configured to use the solver(s) and / or computer-implemented methods described herein.

[0212] In one embodiment, the plurality of impulse responses are pre-processed to generate a plurality of modified impulse responses for training a machine learning model, e.g., as described herein in connection with the above aspects.

[0213] Preprocessing may include, for example, applying a filter to enhance the audio range in the impulse response, such as the range of the average human voice, from 3 kHz to 17 kHz, or from 350 Hz to 17 kHz.

[0214] Other types of preprocessing may also be performed, for example, the impulse responses may be segmented so that only a portion of them is used, e.g., only a predetermined number of reverberations are considered and trained.

[0215] In one embodiment, the method further includes using a plurality of residual audio signals as inputs to provide a plurality of reverberant audio signals. The reverberant audio signals may be, for example, audio signals that include reverberation from a room or space. Thus, the reverberant audio signals include not only raw audio but also information about a space, such as a room, that has passed through, reflected, and absorbed at boundaries within the room. Thus, using the reverberant audio signals as inputs provides substantial information for training a machine learning model.

[0216] For example, in one embodiment, the reverberant audio signal is provided by convolving each of a plurality of impulse responses with a basis audio signal.

[0217] In another embodiment, the reverberant audio signal may be provided by recording an audio signal received by at least one microphone of an audio device.

[0218] As discussed herein, other inputs for training a machine learning model for audio compensation may be, for example, 3D models of multiple rooms, which provide training that may provide specific audio compensation based on the geometry of the rooms, thereby further improving the audio compensation.

[0219] In another embodiment, the method may include receiving a digital model of an audio device and using the digital model of the audio device as input. By using the model, e.g., geometry or other parameters, of the audio device on which the audio compensation is performed, further improved audio compensation may be provided. The audio device may, for example, include additional features or shapes that can identify the direction of sound, which can be used as an advantage for the audio compensation.

[0220] In yet another embodiment, at least one preferred listening position for each of a plurality of rooms may be used as an input for training a machine learning model for audio compensation. In most situations, sounds emitted from an audio device are at the listener's position so that the audio is easily understandable. Thus, by providing the preferred listening position as an input for training a machine learning model, the audio device can provide audio compensation so that the audio is most clearly heard at the preferred listening position, as an example, rather than at the audio device itself.

[0221] The person or audio detection device can be integrated or coupled with an audio device such that the audio device containing the trained model changes its preferred listening position depending on the direction or location of the detected person or audio.

[0222] Also, as discussed, different training steps may be used together or separately to train a machine learning model as described herein.

[0223] For example, in one embodiment, training the machine learning model includes using a plurality of reverberant audio signals described herein as input, and the training includes reestablishing a basis audio signal as output.

[0224] In another or additional embodiment, 3D models of multiple rooms can be used as input, and at least one preferred listening point in each of the multiple rooms is used as input to train the machine learning model. The method may further include the machine learning model re-establishing as output a basis audio signal at the at least one preferred listening point.

[0225] In a further embodiment, training the machine learning model may include generating a compensation impulse response as an output. The compensation impulse response may be based on, for example, the described reverberant audio signal and on basis audio signals as input. The basis audio signals discussed herein may be, for example, anechoic signals, i.e., signals without reverberation information, such as spatial audio.

[0226] In yet a further embodiment, 3D models of a plurality of rooms are used as input, and at least one preferred listening point in each of the plurality of rooms is used as input, and the method further includes generating a compensation impulse response as output at the at least one preferred listening point.

[0227] The impulse responses used to train the machine learning models discussed herein can be generated for each of the 3D models of rooms described herein, i.e., by using a wave-based solver.

[0228] The machine learning models applied herein may, in one embodiment, include neural networks.

[0229] For example, the neural network may include an autoencoder for encoding any of the inputs to the model or for generating compressed inputs for use in training a machine learning model for audio compensation.

[0230] Alternatively, or in addition, the neural network may further include a generative adversarial network (GAN) for generating any one of the inputs for training the machine learning model for audio compensation.

[0231] Additionally, in one embodiment, the neural network may include a deep neural network, a convolutional neural network, and / or a transformer for training a machine learning model for audio compensation.

[0232] In one embodiment, an audio device including at least one microphone may be provided in a method for providing a machine learning model for audio compensation in an audio device, whereby the machine learning model is trained as described above.

[0233] The compensated audio signal may be generated, for example, by convolving the received audio signal with a compensation impulse response, as described herein.

[0234] Further, as discussed in one aspect, an audio device can be provided that includes at least one microphone and that includes a processing system for applying machine learning-based audio compensation.

[0235] The audio device may also include at least one speaker.

[0236] In one embodiment, the audio device may include a communications module configured to transmit the compensated audio signal, which may be transmitted to, for example, the cloud, the internet, and / or a network storage center.

[0237] In further or additional embodiments, the compensated audio signal can be transmitted to a remote audio device, the remote audio device including at least one remote speaker for outputting the compensated audio signal.

[0238] The remote audio device may also include a machine learning model for audio compensation, for example, trained as described herein, and thus emits an audio signal through at least one speaker that applies audio compensation to the audio signal.

[0239] In one aspect, a system for training a machine learning model for audio compensation is disclosed, the system including a computer system having a processor coupled to a memory, the processor configured to receive 3D models of a plurality of rooms, each of the 3D models including at least one sound source and at least one acoustic characteristic, receive a plurality of impulse responses at listening positions in each of the plurality of rooms, and train a machine learning model for audio compensation using at least the plurality of impulse responses as inputs.

[0240] As discussed, one aspect of applying and using acoustic simulation as discussed herein can be to obtain HRTFs for a human head.

[0241] The geometry of the user's head includes the geometry of at least one ear. The geometry of at least one ear may be the primary geometry of interest when determining HRTFs, because sound waves reaching the ear may encounter multiple obstacles within the ear itself before passing through the ear canal to the eardrum. Multiple obstacles may cause absorption and / or reflection of sound waves, which may modify the spectrum of the sound waves before they reach the eardrum.

[0242] Wave-based solvers are based on the discontinuous Galerkin finite element method. The discontinuous Galerkin finite element method has the advantage of being able to handle efficient parallelization and the ability to handle complex geometries that can be generated in either 2D or 3D. Other methods, such as the FDTD method, are efficient in parallel processing but cannot handle complex geometries. FEM, BEM, and FVM can handle complex geometries but cannot perform parallel computing efficiently.

[0243] In one embodiment, the wavelength-based solver determines a simulation of sound propagation within an acoustic frequency range. The acoustic frequency range may be between 20 Hz and 20 kHz, which is the audible frequency range. The audible frequency range may vary from person to person and tends to decrease with age. This acoustic frequency range is determined based on the frequency range of interest when determining the HRTF, which may preferably be audible sound.

[0244] The wave-based solver can determine the simulation of sound propagation within the acoustic frequency range, which in one embodiment may preferably be comprised between 20 Hz and 200 Hz, but may also be comprised between 20 Hz and 100 Hz, or more preferably between 20 Hz and 150 Hz.

[0245] In a second embodiment, the acoustic frequency range may preferably be comprised between 20 Hz and 200 Hz, but also between 150 Hz and 1.5 kHz, or more preferably between 100 Hz and 1 kHz.

[0246] In a third embodiment, the acoustic frequency range may preferably be comprised between 20 Hz and 200 Hz, but also between 20 Hz and 6 kHz, or more preferably between 20 Hz and 10 kHz.

[0247] In one embodiment, simulating sound propagation using a wave-based solver is configured to use a wave-based solver and / or a computer-implemented method for simulating sound propagation as described herein.

[0248] Disclosed herein is a computer-implemented method for determining a head-related transfer function, the method including: receiving a 3D model of a user's head; using a wave-based solver to determine a plurality of wave-based impulse responses from impulses emitted by at least one sound source, each of the plurality of wave-based impulse responses being determined at a corresponding head receiver of a plurality of head receivers; and determining a head-related transfer function (HRTF) of the user's head based on the plurality of wave-based impulse responses received at the plurality of head receivers.

[0249] The 3D model of the user's head may further include a head mesh model, which may be received from an auxiliary computer-implemented method for implementing the mesh. The location of the tympanic membrane may be included in the 3D model of the user's head. For example, the 3D model of the user's head may include an indication of the location of the tympanic membrane on the user's head. Preferably, the 3D model of the user's head may include an indication of the locations of two tympanic membranes on the user's head. By including the location of the tympanic membrane, preferably the two tympanic membranes on the user's head, the computer-implemented method can identify the location of the tympanic membrane, thereby appropriately executing the computer-implemented method for determining a head-related transfer function using the location of the tympanic membrane, since the tympanic membrane is an important parameter for determining a head-related transfer function. An approximation of the location of the tympanic membrane may also be received by the computer-implemented method. The computer-implemented method may further include a method for identifying the exact location of the tympanic membrane when receiving the 3D model of the user's head.

[0250] In one embodiment, the discontinuous Galerkin finite element method (DGFEM) or the spectral element method (SEM), both of which are described herein, may be used.

[0251] In one embodiment, determining the head-related transfer functions further includes obtaining a head mesh model representing the geometry of the user's head. Preferably, the head mesh model may be a 3D head mesh model. The geometry of the user's head may be, for example, a head mesh model representing the geometry of the user's head. Meshing may be understood as a general method of generating models in a digital environment, such as a computer. The mesh may preferably be a discretization of geometry into small, simple shapes. The shapes may be triangles or quadrilaterals in 2D and / or tetrahedrons or hexahedrons in 3D. The mesh density control may determine an appropriate mesh density; a mesh that is too coarse may result in inaccurate results, while an overly fine mesh may increase computational costs and simulation time, or in some cases lead to non-convergence of the simulation using the mesh.

[0252] A mesh can be understood as a polygonal mesh that is a collection of vertices, edges and faces that define the shape of a polyhedral object. Throughout this patent application, the term mesh element can be a face such as a triangle, a quadrilateral, another simple convex polygon, or any other combination thereof.

[0253] In one embodiment, the head mesh model is a curvilinear head mesh model. The head mesh model may include curvilinear mesh elements, which may reduce geometric distortions that occur when meshing curved boundaries into discrete mesh elements. To obtain the head mesh model, mesh generation may be performed. Mesh generation is the practice of creating a mesh, which is the subdivision of a continuous geometric space into discrete geometric and topological cells. Mesh generation may generally be performed with linear (affine) mesh elements. The use of linear mesh elements may not adequately represent the boundaries of curved domains in the mesh model unless a very fine mesh is used. An overly fine mesh may lead to undesirably high computational costs, and in some cases, it may be impossible to perform a converged simulation while using a fine mesh. Utilizing curvilinear mesh elements may enable the use of large mesh elements with a high base degree while still capturing important geometric details.

[0254] In particular, curvilinear meshes have been shown to be efficient for simulating sound propagation using wave-based simulations. Curved sides and edges provide high-fidelity results. This is particularly useful, along with the possibility of using wave-based solvers on larger meshes that apply finite element methods (FEMs), such as the discontinuous Galerkin finite element method (DGFEM), higher-order discontinuous Galerkin finite element method, or the spectral element method (SEM), as disclosed herein. In particular, DGFEM and SEM have been shown to be specific finite element methods that are particularly effective for handling curved mesh elements.

[0255] In particular, DGFEM and SEM have been shown to be specific finite element methods that are particularly effective at dealing with curvilinear mesh elements. Curvilinear coordinates are a coordinate system in Euclidean space in which the coordinate lines may be curved. These coordinates can be derived from a set of Cartesian coordinates by using a transformation that is locally invertible at each point. Well-known examples of curvilinear coordinate systems in three-dimensional Euclidean space are cylindrical coordinates and spherical coordinates.

[0256] Obtaining a 3D model of a user's head can be done in various ways. To obtain a 3D model, the user's head may be scanned, for example. The 3D model may be digitally obtained by modeling the 3D model using, for example, CAD (computer-aided design) software. The 3D model may also be obtained by loading a file, for example, an STL file, which is a common format for storing digital models, into a computer. The 3D model may also be obtained by capturing a 3D image of the user's head with a camera or any device capable of capturing an image, preferably a 3D image.

[0257] In another embodiment, generating the head-related transfer functions includes arranging a digital representation of a head array including a plurality of digital representations of head receivers, such as a head mesh model, around a 3D model of the user's head, the head array being arranged such that the distance between any of the digital representations of the head receivers and the head mesh model is no less than a predetermined distance.

[0258] In a preferred embodiment, the 3D model of the user's head further comprises receiving the location of at least one sound source representing the eardrum or a vicinity thereof in the 3D model.

[0259] In another embodiment, the computer-implemented method further includes determining a first closest mesh element on the head mesh model that is closest to the tympanic membrane. As described herein, the location of the tympanic membrane may be received when receiving the 3D model of the user's head. This allows spatial correlation between the first closest mesh element and the location of the tympanic membrane to be performed.

[0260] In one embodiment, the shape of the head array including the digital representations of the head receivers is a sphere. Alternatively, the shape may be an offset shape in which the digital representations of the head receivers are placed / arranged at a predetermined offset distance from the head mesh model. The predetermined offset distance may be the same as the predetermined distance discussed herein.

[0261] The predetermined distance may include, for example, the radius of the head array plus an additional distance to properly encompass the head mesh model, which may be determined, for example, such that the distance from any head receiver on the head array to any point on the 3D model of the user's head is preferably no less than the predetermined distance.

[0262] In one embodiment, the predetermined distance is between 0.5 and 1.5 meters, preferably between 0.8 and 1.2 meters, or most preferably 1 meter. Such a predetermined distance has been shown to be a good choice for simulating a typical room, or preferably for generating, determining, and / or simulating head-related transfer functions.

[0263] The predetermined distance can be used to determine the Ambisonics order N, for example, if this is used for encoding and decoding. For example, the Ambisonics order N can be determined using the formula N>2*pi*f*R(array) / c, where "f" is the frequency under consideration and "c" is the speed of sound. The speed of sound is typically 343 m / s, but the speed of sound may depend on the temperature and the medium in which the sound waves propagate. In that case, R(array) may be the predetermined distance. Determining the minimum order N can then be used to find the (N+1) 2The number of head receivers in the head array can be selected as determined by: This can be multiplied by a factor such as 1.5 or 2.0 to obtain higher fidelity, but at the cost of increased simulation time.

[0264] When applying a wave-based solver to generate head-related transfer functions, the number of sound sources generally determines the time and resources required to determine the acoustic simulation. Thus, if the first nearest mesh element representing the eardrum is set to function as a microphone, all receivers in the head array must function as sound sources and emit impulse signals. However, each of these must be solved individually, and the time and resources used for the solution increase significantly based on the number of sound sources in the head array. Instead, to determine the head-related transfer functions discussed herein, a first impulse signal can be advantageously emitted using an iterative law, using the eardrum as the sound source.

[0265] In one embodiment, where the 3D model of the user's head is a head mesh model, a first closest mesh element that is closest to the eardrum or the location of the eardrum on the 3D model of the user's head can be determined. The first closest mesh element, in one embodiment, can be used as the sound source for emitting the first impulse signal. By having and / or identifying the first closest mesh element that is closest to the eardrum on the 3D model of the user's head, the sound source for emitting the first impulse signal can be set.

[0266] Therefore, based on the emitted first impulse response, multiple first impulse responses can be determined using a wave-based solver, and each of the first impulse responses can describe the impulse response of the first impulse signal received at a respective head receiver.

[0267] Therefore, the head-related transfer function of the eardrum can be generated by combining a plurality of first impulse responses.

[0268] The first impulse signal should preferably be a perfect impulse and have a flat spectrum, however, this may not be possible and in order to increase the fidelity of the head-related transfer function, the first impulse response should preferably be source corrected by a reference signal.

[0269] In one embodiment, this can be done by arranging a digital representation of a first source-compensated microphone located at a first source distance from a first nearest mesh element, the first source distance being less than a predetermined distance.

[0270] Thus, a first source correction signal may be determined using a wave-based solver, the first source correction signal describing a first impulse signal received at a first source correction microphone.

[0271] A plurality of first source-corrected head impulse responses can then be determined by source correcting each of the plurality of wave-based impulse responses using the first source-corrected signal, and a first head-related transfer function of the user's head for the first eardrum can then be determined by combining the plurality of first source-corrected head impulse responses.

[0272] In one embodiment, generating the head-related transfer function comprises determining an energy content of at least one frequency of the first head-related transfer function.

[0273] In one embodiment, determining the energy content of at least one frequency of the head-related transfer function comprises determining different Ambisonics orders corresponding to different levels of energy content.

[0274] In one embodiment, determining the energy content of at least one frequency of the head-related transfer function comprises determining an Ambisonics order N for the energy content of the at least one frequency.

[0275] In a further embodiment, the Ambisonics order N is determined for a number of frequencies and the energy content of each frequency is determined.

[0276] In a further embodiment, determining the Ambisonics order N of the energy content is based on determining the energy content as a sum of the Ambisonics coefficients of each order N, and then normalizing to 1 for each frequency.

[0277] In one embodiment, the energy content is determined for a frequency range such as 0-20 kHz, 0-10 kHz, 10-20 kHz, 0-9 kHz, 0-8 kHz, 0-7 kHz, 0-6 kHz, etc., 0-5 kHz, 0-4 kHz, 0-3 kHz, 0-2 kHz, 0-1 kHz, etc. Preferably, the frequency range may be included in the audible spectrum. The maximum frequency of this range may dictate the Ambisonics order N to be selected. In general, the higher the maximum frequency, the higher the Ambisonics order N should be.

[0278] In one embodiment, the 3D model of the user's head includes multiple tympanic membranes, such as two tympanic membranes. In such a case, the computer-implemented method discussed herein is repeated for each tympanic membrane. In other words, each of the multiple tympanic membranes may be treated as a tympanic membrane such that multiple head-related transfer functions are generated for that tympanic membrane.

[0279] In one embodiment, determining the head related transfer function comprises: obtaining a 3D box model including a highly acoustically absorbing surface or a 3D box model having a predefined size, wherein a first impulse signal received from a sound source at each of a plurality of digital representations of the head receivers does not include reflections caused by the surface of the 3D box model; -The head receiver array and head mesh model are arranged in a 3D box model.

[0280] By obtaining a 3D box model including a highly acoustically absorbing surface, the first impulse signal emanating from the first nearest mesh element is not reflected by the surface of the 3D box model and is therefore not received by the digital representation of the head receiver array, which includes the multiple digital representations of the head receiver array. Preferably, the multiple digital representations of the head receivers can receive the incoming impulse signal, preferably without receiving reflections caused by any surfaces, obstacles, or geometry external to the digital representation of the head receiver array. In another embodiment, the 3D box model has a predefined size, and the first impulse signal received from the sound source to each of the multiple digital representations of the head receivers does not include reflections caused by the surface of the 3D box model. The predefined size can be estimated such that the surface is far away from the multiple digital representations of the head receivers. A predefined size that is too large can be costly in terms of computational cost and time, and therefore the predefined size needs to be estimated and / or calculated so that the simulation of impulse signal propagation can be stopped before the impulse signal reaches the surface of the 3D box model, or substantially before it reaches the surface of the 3D box model.

[0281] Alternatively, the 3D box model may be a 3D spherical model, which may preferably be constructed from a sphere.

[0282] The invention will now be described in more detail with reference to the accompanying drawings. [Brief explanation of the drawings]

[0283] [Figure 1] 1 shows a schematic diagram of an embodiment of a simulation and simple rendering of acoustic wave propagation as described herein. [Figure 2] 10A-10C show schematic diagrams of different embodiments of boundary models used to describe the acoustic properties of materials used in the simulations. [Figure 3]10A-10C show schematic diagrams of different embodiments of boundary models used to describe the acoustic properties of materials used in the simulations. [Figure 4] 1 shows a diagram of an embodiment of a method used to spatially render signals from a wave-based simulation. [Figure 5] 1 illustrates a system according to an embodiment of the present invention.

[0284] Detailed Description of the Drawings 1, a simulation 100 includes an input step 101 and a preprocessing step 102. The simulation then branches into a DG (Discontinuous Galerkin Finite Element Method) module 120, which implements an embodiment of the Finite Element Method (FEM) described herein for simulating wave propagation at low to mid frequencies, and a GA (Geometric Acoustics) module 130, which implements an embodiment of the Geometric Acoustics Solver described herein for simulating wave propagation at high frequencies.

[0285] The input, in this embodiment, is a 3D model of a virtual space 104 for which simulation is desired. As shown in FIG. 1 , the virtual space is defined by a floor 105, four walls 106, 107, 108, and 109, and a ceiling 110, so that the model is watertight. The outer boundaries defining the virtual space, i.e., the walls, ceiling, and floor, are shown as transparent elements to show the contents of the virtual space. The 3D model of the virtual space further includes a window 111 arranged on one of the walls 106, two tables 112 and 113, a room divider 114 arranged between the two tables 112 and 113, and a sound-absorbing ceiling tile 115 suspended from the ceiling 110. The different elements of the virtual space are described by their 3D shapes in the 3D model, but also by their respective acoustic properties, which will be described below. In other words, each surface describing a different element defines the boundary of the model, and the boundary properties describe how the simulated sound is affected when interacting with the boundary.

[0286] A sound source 116 is positioned on one of the tables 112. In this embodiment, the sound source is omnidirectional, i.e., sound propagates in all directions. In other embodiments, the sound source may be directional, for example.

[0287] During pre-processing, different data may be prepared for use in the simulation. For example, in this embodiment, the 3D model of the virtual space is converted into a curvilinear mesh model. A curvilinear mesh model has the advantage that sound reflecting from rounded and curved surfaces is more appropriately simulated. Typically, curved surfaces have been represented with a staircase model, in which horizontal and vertical surfaces approximate the curvature. From a modeling perspective, this can be good (e.g., at high resolution). However, when simulating acoustic waves, this always creates either vertical or horizontal surfaces, which reflect sound, and even high resolution cannot correct this. Therefore, by generating a curvilinear mesh model, boundary regions and surfaces can be simulated in more detail for curved and bent objects.

[0288] Similarly, a 3D model of a virtual space is described as a volumetric 3D model, where the virtual space defining the 3D model is divided / defined into a discretized volumetric mesh or subdomains that provide the discretized volumetric mesh. In this embodiment, the volumetric mesh is composed of an unstructured mesh of tetrahedral elements, and the discretization process processes the 3D model so that the tetrahedrons are as uniform as possible. As described below, this advantageously improves computation time when applying the DG method to the 3D model. As mentioned above, other element shapes can also be used.

[0289] Once the inputs are received and prepared, the simulation splits the simulation into two parts: a simulation at low and mid frequencies using a wave-based solver in the DG module 120, and a simulation at high frequencies using a geometric acoustic solver in the GA module 130.

[0290] A DG module 120 excites the volumetric model with an impulse 121 at the source location. Once the impulse is excited in the volumetric model, a DG method 122 is applied to a discretized mesh and boundary region module 123, and a time marching module 124 is used to simulate the wave propagation of the impulse in the 3D model.

[0291] Applying the DG method to a discretized mesh has the advantage that the DG method can be applied separately and independently to each volume of a 3D model, allowing the method to be calculated for each volume in parallel. This allows wave-based simulations to be performed while significantly reducing computation time. In particular, this has proven to be an advantageous aspect of finite element methods (FEMs), such as the discontinuous Galerkin finite element method, since they can process each volume independently and thereby in parallel with each other, significantly improving processing time.

[0292] One important aspect of wave-based simulation is how wave propagation reacts at boundaries. A boundary should be understood as any change in material or matter across which waves travel or which waves impinge or encounter. For example, a boundary region can be, for example, a surface that reflects sound, a surface or material that absorbs sound, or a surface that diffuses sound.

[0293] When applying DG to the volumetric model 122, the boundary region module 123 maintains a model that describes the acoustic properties of different materials within the 3D model of the virtual space.

[0294] Similarly, applying DG methods separately to a volumetric 3D model results in a simulation that shows only the results at a particular point in time. Therefore, to provide a temporal simulation, a time-marching module 124 is provided. The time-marching module includes one or more time-marching algorithms that connect temporal aspects with aspects of the DG method over time. Implementations of time-marching algorithms have been described previously in this specification.

[0295] The time marching module tells the system how the wave-based propagation is simulated over time, causing the system to repeat steps 122, 123, and 124 for the entire duration of the simulation, i.e., until the wave-based simulation is complete.

[0296] The duration of the simulation can be determined in different ways. One way is to simply decide to run the simulation for a predetermined period of time. However, this is most likely to produce a redundant simulation, as the time will most likely be set to ensure that the simulation is completed. For example, if an impulse is simulated to determine an impulse response, the time will be set to ensure that all relevant information is obtained in relation to the impulse response.

[0297] However, in this embodiment, the global energy in the virtual space is determined each time the iteration reaches step 124 , and if the global energy level is below a certain threshold, the method proceeds to step 125 .

[0298] When DG is applied to a volumetric model as described, the result is a simulation of the wave propagation of an impulse from a sound source location throughout the 3D model of the virtual space. Thus, for any listening position in the virtual space, the impulse response can be derived (125), for example, in table 113 indicated by listening point location 117, since the simulation can be played back while recording the impulse response of the sound source location at that listening point.

[0299] At higher frequencies, the simulation is performed in the GA module 130. As mentioned above, the advantage of using geometric acoustics at higher frequencies is that wave propagation is treated as ray propagation, which allows the use of ray tracing, significantly reducing processing time for high frequencies, and diffraction, diffusion, and other wave-based principles do not have the same effect as at lower frequencies and can therefore be considered negligible. For example, depending on the application, the use and type of rendering in which the simulation is used for the frequency intervals defining the low and respective high frequencies may differ and be selected to provide the best results and desired speed in a particular case.

[0300] Since the geometric acoustics solver (GA module) considers the energy decay of sound propagation, the model is not excited with an impulse as in the DG module. Instead, decay information is derived and used to establish the impulse response. Simulation in this embodiment is performed using image source processing in image source module 132, followed by a ray tracing algorithm in ray tracing module 133.

[0301] Both image source processing and ray tracing algorithms estimate sound propagation as ray reflections, e.g., specular reflections. However, image source processing is more accurate because it determines exact specular reflections, while ray tracing is less accurate because it is a probabilistic method. Therefore, in this embodiment, the image source is used for the first combination of ray reflections for high-energy rays, and the ray tracing method is used for the remaining reflections until the simulation is stopped.

[0302] The results from the image source simulation and the ray tracing simulation are then hybridized into one simulation in the hybridization module 134, so that a simulation of wave propagation of impulses at high frequencies can be established.

[0303] Similar to the DG module step of deriving the impulse response 125 , the impulse response of the GA method is derived in step 135 .

[0304] The DG impulse response and the GA impulse response are merged or hybridized into one simulated impulse response ( 136 ).

[0305] The simulated impulse response can be used to render sound (140). As mentioned above, rendering sound can be done in many different ways. In this embodiment, one of the simplest methods is shown, in which the simulated impulse response is convolved with the source audio (141). This is a common way to provide a soundscape for a sound file, thus providing an audio rendering (142) that creates the audible experience of standing at the location 117 where the impulse response was registered and listening to the audio coming from the source location.

[0306] Figures 2 and 3 show two different embodiments of the boundary absorption model, which can be used to describe the acoustic properties of a material when an acoustic wave impinges on its surface / boundary.

[0307] 2 shows a boundary surface model in which an acoustic wave 200 emitted from an acoustic source 201 travels through a material 202 and impinges on a second material 203. The direct acoustic wave 200' is shown by a dashed circle segment, and the first reflections 200'' of the acoustic wave are shown by dashed circle segments with a greater number of dashed segments.

[0308] The boundary surface model is considered in this embodiment at two points A and B on the surface of the second material 203. However, it should be understood that the incident sound wave 200 impinges on the surface of the second material over the entire surface facing the sound source 201. Therefore, a similar consideration can be made for any point on the surface of the second material.

[0309] At point A, a first reflection occurs at an angle of incidence α along a first incident direction 204. The sound wave is reflected off the surface along a first reflection direction 205. A portion of the energy is absorbed by the second material, as indicated by arrow 206.

[0310] Similarly, at point B, a first reflection occurs at an angle of incidence α along a second direction of incidence 207. As can be seen, the angle of incidence β is different from the angle of incidence α. The sound wave at point B is reflected from the surface along a second direction of reflection 208, and a portion of the energy is absorbed by the second material, as indicated by arrow 209.

[0311] In this embodiment, a boundary absorption model is used, where the reflected sound 205, 208 is the same and does not depend on the angle of incidence, and therefore the absorbed sound and energy 206, 209 is the same.

[0312] Such an absorption model is suitable for highly reflective materials, for example hard smooth surfaces such as glass, brick, and metal.

[0313] The same setup, but with a different second material 303, is shown in Figure 3, where the energy absorbed and reflected varies depending on the angle of the incident sound wave at a particular point on the surface.

[0314] Similar to Figure 2, Figure 3 shows an acoustic wave 300 emanating from a source 301, traveling through a material 302, and impinging on a second material 303. The direct acoustic wave 300' is indicated by a dashed circle segment, and the first reflections 300'' of the acoustic wave are indicated by a dashed circle segment with a greater number of dashed segments.

[0315] The boundary surface model is considered in this embodiment at two points A and B on the surface of the second material 303. However, it should be understood that the incident sound wave 300 impinges on the surface of the second material over the entire surface facing the sound source 301. Therefore, similar considerations can be made for any point on the surface of the second material.

[0316] At point A, a first reflection occurs at an angle of incidence α along a first incident direction 304. The sound wave is reflected off the surface along a first reflection direction 305. A portion of the energy is absorbed by the second material, as indicated by arrow 306.

[0317] Similarly, at point B, a first reflection occurs at an angle of incidence β along a second direction of incidence 307. As can be seen, the angle of incidence β is different from the angle of incidence α. The sound wave at point B is reflected from the surface along a second direction of reflection 308, and a portion of the energy is absorbed by the second material, as indicated by arrow 309.

[0318] In this embodiment, a boundary absorption model is used, where the reflected sounds 305, 308 are different and depend on the angle of incidence, and therefore the absorbed sounds and energies 306, 309 are different as well.

[0319] This property is found, for example, in the surfaces of porous materials such as many sound-insulating materials, because uneven surfaces disrupt and scatter the reflections, especially at low angles of incidence of sound waves.

[0320] Also, some materials may propagate waves further, for example, if a sound wave first travels through air and hits a body of water, the transition between air and water is described by the boundary absorption model as described above, but water propagates further and the sound wave has different properties than air, for example, sound in water travels faster and loses less energy than sound in air.

[0321] Because the above model focuses on accounting for absorption and reflection at the boundary between two materials, there is also energy loss not shown in the above model that is typically considered in the simulation, which depends on the distance traveled by the sound wave. Thus, the energy of the direct sound wave at point B in the above embodiment will typically be less than the energy of the direct sound wave at point A, because the sound wave has traveled a longer distance to reach point B.

[0322] FIG. 4 shows a diagram of an embodiment of a method used to spatially render signals from a wave-based simulation. Preferably, a wideband wave-based acoustic framework is shown in FIG. 4, where a wideband wave-based acoustic framework can be used to obtain a spatial room impulse response. The spatial room impulse response can then be encoded into spherical harmonic contributions 402, resulting in a potentially higher-order Ambisonics formulation, which can be combined with free-field head-related transfer functions (HRTFs) 403, 405 for binaural rendering 407. In this embodiment, binaural rendering or binaural auralization can be understood as the combination of transfer functions and impulse responses, e.g., a spatial impulse response generated for a particular room at listening and HRTFs. The echo-free signal may be convolved with the binaural auralization (not shown) to generate a rendered sound at the listening point.

[0323] The wideband wave-based acoustics framework may also be used for multi-channel rendering, in which higher-order Ambisonics formulations may be combined with transfer functions that include more than two channels. In other words, while HRTFs include two receivers or two channels, other transfer functions may include more than two receivers or channels and may be used in the wideband wave-based acoustics framework, as shown in FIG. 4, thereby enabling multi-channel rendering.

[0324] Wave-based simulation The wave-based simulation 401 that may be used in the embodiment shown in FIG. 4 may be a wave-based simulation as described herein. Preferably, the wave-based simulation 401 may be a discontinuous Galerkin finite element method, which may also be described as a discontinuous Galerkin method. Preferably, the wave-based simulation may simulate acoustic waves over a wide frequency range, such as a first frequency range, e.g., 20 Hz to 20 kHz, e.g., 20 Hz to 10 kHz, e.g., 20 Hz to 8 Hz, e.g., 50 Hz to 8 kHz, e.g., 100 Hz to 8 kHz. As described herein, the discontinuous Galerkin finite element method may have the advantage of being able to handle efficient parallelization and the ability to handle complex geometries that may be generated in either 2D or 3D.

[0325] As shown in Figure 4, a wave-based simulation 401 is first performed using one or several virtual spherical microphone arrays 402 embedded in a 3D model of a domain centered at a given location. The 3D model of the domain may also include at least one sound source and acoustic properties of multiple boundaries. The wave-based simulation 401 can simulate a sound field from at least one sound source into the 3D model of the domain. The given location will preferably be a listening point.

[0326] The wave-based simulation may be performed by a wave-based solver, as described herein, that may determine a wave impulse response of a wave-based propagation of an impulse emitted from at least one sound source within the 3D model of the domain and received at a listening point within a first frequency range.

[0327] Spatial rendering and further processing Wave Amplitude Density

number

number

[18] . The wave amplitude densities can be processed to enable binaural auralization, for example by reconstructing the sound field that would have reached the eardrums of a virtual listener embedded in a virtual room and preferably located at a listening point or a given location in the virtual room. Such spatial rendering allows the separation of different wave contributions according to their incidence direction.

number

number

number

number

number

number

number

[0328] Equation (14) is first

number

number

number

number

number

number

number

number

number

number

number

number

number

[0329] The ability to spatially render simulation data may require an estimate of wave amplitude density.

[0330] Receiver Array and Processing As shown in Figure 4, an array of secondary receivers 402 can be placed around a sampling location or listening point to encode the sound field. The secondary receivers can be microphones.

number

number

number

number

number

number

number

number

number

[0331] Taking the inverse Fourier transform of (18) directly gives estimates of the Ambisonics coefficients as follows:

number

number

number

number

number

[0332] Another solution to improve the robustness of the spatial encoding procedure can consist of an open spherical array of directional receivers with a first-order cardioid directivity pattern. In the latter case, it can be shown that the radiation function is

number

number

number

number

number

number

[0333] Number of receivers in the array

number

number

number

number

number

number

number

[0334] FIG. 4 shows a diagram of an embodiment of a method used to spatially render an acoustic signal from a wave-based simulation. Once the wave-based simulation is complete, the results are post-processed to extract wave amplitude density coefficients by solving equation (19) based on projections to spherical harmonic basis functions. The spatial rendering equation (16), or equivalently, (17), can be evaluated to account for a given binaural HRTF dataset (given orientation) in the wave-based simulation. The spatial rendering can then be obtained by convolving the binaural HRTF dataset in the wave-based simulation with the audio signal. Multiple balloon plots are shown in the center of the figure,

number

[0335] The HRTF dataset 405 may include different orientation data. Preferably, the HRTF dataset may be simulated or measured using an HRTF reference. The HRTF reference may not correlate with the Ambisonics coefficients, which may result in a mismatch with the Ambisonics coefficient reference when solving the spatial rendering equations. This may require a rotation 406 of the HRTF reference to align the HRTF reference with the Ambisonics coefficient reference. If the HRTF reference does not align with the Ambisonics coefficient reference, the rendered signal may be generated with an offset, which may result in an incorrect audio rendering experience for the user.

[0336] In parallel with the wave-based simulation and binaural auralization as described, a geometrical acoustic-based simulation may also be performed for the second frequency range. Thus, the final result may be a hybridization of the geometrical acoustic-based simulation and the wave-based simulation. The hybridization may be, for example, a hybridization as described in this disclosure.

[0337] FIG. 5 illustrates a system 500 according to an embodiment of the present invention. The system 500 includes a computer system 502. The computer system 502 includes dedicated hardware and / or software modules executing on a processor 504 coupled to a memory 506. The computer system 502 may also be communicatively coupled to a communications network 508. The network 508 may be a public network, such as the Internet, or a private network, such as an intranet. The network 508 may also be a combination of public and / or private networks. The computer system 502 may be directly coupled to the network 508, for example, via an Ethernet cable or via a wireless connection such as Wi-Fi. The computer system 502 may also be indirectly coupled to the network 508 in other ways known to those skilled in the art, for example, but not limited to, via another device (not shown), such as a router, a switch, a hub, a separate computer system, a mobile device, a modem, and / or a combination of these devices. The processor 504 is configured to perform any of the methods described in detail throughout this disclosure.

[0338] The computer system 502 further includes one or both of an impulse response generation module 512, a neural network training module 514, and an HRTF generation module 524, all of which execute on the processor 504. While all modules are shown in FIG. 5 , it should be explicitly noted that only the impulse response generation model 512, or only the neural network training module 514 or the HRTF generation module 524 may be present. The impulse response generation module 512 is configured to perform one or more methods for generating impulse responses, as described in detail above. Illustratively, the impulse response generation module may also include at least one of a wave-based solver and a geometric audio-based solver. The neural network training module 514 is configured to perform one or more methods for training a machine learning model for audio compensation, as described in detail above. The HRTF generation module 524 is configured to perform one or more methods for generating HRTFs, as described in detail above. In some embodiments, modules 512, 514, and 524 may be specialized sets of computer software instructions programmed on one or more dedicated processors in computer system 502 and may include memory locations and / or registers specifically designed for executing the specialized computer software instructions.

[0339] 5 as executing within the same computer system 502, it should be expressly noted that the functionality of modules 512, 514, and 524 may be distributed among multiple computer systems. Computer system 502 enables modules 512, 514, and 524 to communicate with each other to exchange data for purposes of performing the described functions. It should be understood that any number of computing devices arranged in various architectures, resources, and configurations (e.g., cluster computing, virtual computing, cloud computing) may be used without departing from the scope of the present invention. Exemplary functionality of modules 512, 514, and 524 is described in detail throughout this specification.

[0340] In some embodiments, the machine learning model 510 is coupled to the network 508 or included in the computer system 502, as shown in FIG. 5 . The machine learning model 510 may be a single architecture or a combination of multiple neural network architectures. For example, the machine learning model 510 may include a first neural network architecture and a second neural network architecture, which are distinct entities. The machine learning model 510 is configured to provide and / or perform the functions described in detail above. The machine learning model 510 may be coupled to the network 508 as shown and may communicate with the computer system 502 via the network 508, although it is expressly contemplated that the machine learning model may be part of the computer system 502. The impulse response generation module 512 and / or the neural network training module 514 and / or the HRTF generation module 524 communicate with the machine learning model 510 to exchange data for purposes of performing the described functions. It should also be expressly noted that the functionality of the machine learning model 510 may be distributed among multiple computer systems. Similar to those described above with reference to modules 512, 514, and 524, any number of computing devices arranged in a variety of architectures, resources, and configurations may be used without departing from the scope of the present invention.

[0341] In some embodiments, audio device 516 is coupled to network 508. Audio device 516 includes one or both of a microphone 518 and a speaker 520. Audio device 516 may also include a processing system 522. Audio device 516 and / or its processing system 522 may be configured to apply machine learning-based audio compensation, as described in detail above. To this end, audio device 516 and / or processing system 522 may be communicatively coupled to a machine learning model, such as machine learning model 510, to send data to and receive data from the model. Illustratively, audio device 516 is shown in FIG. 5 as coupled to network 508, e.g., by using a communications module. However, it is expressly contemplated that audio device 516 may not be coupled to network 508. Illustratively, audio device 516 may include a machine learning model substantially similar to machine learning model 510, configured to apply machine learning-based audio compensation, as described in detail above. It should be noted that although only one microphone 518 and one speaker 520 are shown in FIG. 5, audio device 516 may include multiple microphones and / or multiple speakers.

[0342] Reference list:

[0343] [1] F. Pend, APEngsig-Karup, C.-H. Jeong, JSHesthaven, MSMejling, and J. Strettimann-Andersen. “Time domain room acoustic simulations using the spectral element method”. In: The Journal of the Acoustical Society of America 145.6 (2019), pp. 3299-3310.

[0344] [2]G.Gabard and E.J.Brambley.“A full discrete dispersion analysis of time-domain simulations of acoustic liners with flow”. In: Journal of Computational Physics 273(2014),pp.310‐326.

[0345] [3]D.Dragna,P.Pineau,and P.Blanc-Benon.“A generalized recursive convolution method for time-domain propagation in porous media”.In:The Journal of the Acoustical Society of America 138.2(2015),pp.1030‐1042.

[0346] [4]H.Wang and M.Hornikx.“Time-domain impedance boundary condition modeling with the discontinuous Galerkin method for room acoustics simulations”.In:The Journal of the Acoustical Society of America 147.4(2020),pp.2534‐2546.

[0347] [5]M.Cosnefroy.“Propagation of impulsive sounds in the atmosphere:numerical simulations and comparison with experiments”.Partly in French.PhD thesis.Ecole Centrale de Lyon,2019.

[0348] [6]F.Pind,C.-H.Jeong,A.P.Engsig-Karup,J.S.Hesthaven,and J.Stroann-Andersen.“Time-domain room acoustic simulations with extended-reacting porous absorbers using the discontinuous Galerkin method”.In:The Journal of the Acoustical Society of America 148.5(2020),pp.2851‐2863.

[0349] [7]F.Pind,C.-H.Jeong,J.S.Hesthaven,A.P.Engsig-Karup,and J.Stromann-Andersen.“A phenomenological extended-reaction boundary model for time-domain wave-based acoustic simulations under sparse reflection conditions using a wave splitting method”.In:Applied Acoustics 172 (2021),p.107596.

[0350] [8]T.W.J.S.Hesthaven.Nodal Discontinuous Galerkin Methods.Springer-Verlag GmbH,2007.

[0351] [9]H.L.Atkins and C.-W.Shu.“Quadrature-Free Implementation of Discontinuous Galerkin Method for Hyperbolic Equations”.In:AIAA Journal 36.5(1998),pp.775‐782.

[0352]

[10] A.Jameson and T.Baker.“Solution of the Euler equations for complex configurations”.In:6th Computational Fluid Dynamics Conference.American Institute of Aeronautics and Astronautics(AIAA),1983.

[0353]

[11] F.Q.Hu.“Low-dissipation and low-dispersion Runge-Kutta schemes for computational acoustics”.In:Journal of Computational Physics 124.1(1996),pp.177‐191.

[0354]

[12] J.Berland,C.Bogey,and C.Bailly.“Low-dissipation and low-dispersion fourth-order Runge-Kutta algorithm”.In:Computers & Fluids 35.10(2006),pp.1459‐1463.

[0355]

[13] H.Wang,M.Cosnefroy, and M.Hornikx.“An arbitrary high-order discontinuous Galerkin method with local time-stepping for linear acoustic wave propagation”.In:The Journal of the Acoustical Society of America 149.1(2021),pp.569‐580.

[0356]

[14] A.Melander and E.Strom.“Massively Parallel Nodal Discontinous Galerkin Finite Element Method Simulator for Room Acoustics”.MA thesis.Technical University of Denmark,2020.

[0357]

[15] J.S.Hesthaven and T. Warburton,Nodal Discontinuous Galerkin Methods‐Algorithms,Analysis,and Applications(Springer,New York,2008),Chap.3.

[0358]

[16] H.Kuttruff,“Room Acoustics: 6 th edition”,December 10,2019,CRC Press.

[0359]

[17] F.Pind,“Wave-based Virtual Acoustics”,2020,Technical University of Denmark - https: / / orbit.dtu.dk / en / publications / wave-based-virtual-acoustics.

[0360]

[18] S.Bilbao,A.Politis,and B.Hamilton.“Local time-domain spherical harmonic spatial encoding for wave-based acoustic simulation”.IEEE Signal Processing Letters 26.4 (2019),pp.617-621.

[0361]

[19] L.Savioja and UPSvensson.Overview of geometrical room acoustic modeling techniques.J.Acoust.Soc.Am.,138:708-730,2015.

[0362] Further details of the invention 1. A wave-based solver for simulating the propagation of sound within at least one model of a virtual domain based on at least one sound source within the virtual domain and at least one acoustic property of the virtual domain, the wave-based solver including a finite element method (FEM) such as a continuous Galerkin finite element method (DGFEM) or a spectral element method (SEM).

[0363] 2. The wave-based solver of clause 1, wherein the virtual domain is a one-dimensional (1D) domain, a two-dimensional (2D) domain, or a three-dimensional (3D) domain such as a three-dimensional model (3D) of a virtual space.

[0364] 3. The wave-based solver of clause 1, wherein the 3D model of the virtual space defines a bounding surface of an air volume.

[0365] 4. A wave-based solver according to clause 1 or 2, wherein the boundary surface is a mesh, and thus the 3D model of the virtual space is represented as a 3D mesh model.

[0366] 5. The wave-based solver of clause 4, wherein the 3D mesh model consists of curves.

[0367] 6. The wave-based solver of any one of clauses 3 to 5, wherein the air volume is represented as a volumetric 3D mesh model.

[0368] 7. The wave-based solver of clause 6, wherein a qualitative assessment is provided for the volumetric 3D mesh model to assess critical areas of sound propagation.

[0369] 8. A wave-based solver according to any one of clauses 1 to 7, wherein the virtual space is numerically discretized into subdomains using an acoustic wave equation module.

[0370] 9. The wave-based solver of any one of clauses 2 to 8, wherein the volumetric 3D mesh model of the air volume represents a discretized subdomain.

[0371] 10. The wave-based solver of clause 8 or 9, wherein the FEM is applied separately to each subdomain.

[0372] 11. A wave-based solver described in any one of clauses 1 to 10, wherein the wave-based solver includes a locally responsive frequency-dependent boundary modeling sub-module for simulating the propagation of the sound in at least one first boundary region including a first material.

[0373] 12. A wave-based solver described in any one of clauses 1 to 11, wherein the wave-based solver includes an extended reaction frequency sub-module for simulating the propagation of the sound in at least one second boundary region including a second material.

[0374] 13. The wave-based solver of clause 12, wherein the extended reactive frequency-dependent boundary modeling submodule is configured to model the second material in a fluid-structure-acoustic simulation.

[0375] 14. A wave-based solver according to any one of clauses 12 to 13, wherein the second material is a porous material.

[0376] 15. A wave-based solver as described in any one of clauses 1 to 14, wherein the 3D model includes at least one directional sound source for emitting sound in a defined direction.

[0377] 16. A wave-based solver as described in any one of clauses 1 to 15, wherein the wave-based solver includes a time-marching sub-module including at least a first time-marching method for simulating the propagation of the sound in time and space.

[0378] 17. The wave-based solver of clause 16, wherein the time-marching sub-module includes a second time-marching method for simulating the propagation of sound in time and space, the first time-marching method including a standard low-volume explicit Runge-Kutta algorithm, and the second time-marching method including an implicit-explicit time-marching algorithm.

[0379] 18. The wave-based solver of any one of clauses 1-17, wherein each subdomain is solved independently on a central processing unit (CPU) and / or a graphics processing unit (GPU).

[0380] 19. The wave-based solver of any one of clauses 1 to 18, wherein a message passing interface (MPI) is configured to handle communication between central processing units (CPUs).

[0381] 20. The wave-based solver of clause 19, wherein the communication processed by the MPI is performed following a halo exchange.

[0382] 21. A wave-based solver described in any one of clauses 1 to 20, wherein the wave-based solver further comprises extracting one or more wave impulse response(s) based on the simulation of the propagation of the sound.

[0383] 22. The wave-based solver of clause 21, wherein the wave impulse response(s) are spatial impulse response(s).

[0384] 23. A wave-based solver according to any one of clauses 1 to 22, wherein the wave-based solver renders the simulation of the sound propagation to one or more output format(s), such as visual and / or graphical and / or numerical, and / or audible output format(s).

[0385] 24. The wave-based solver of clause 23, wherein the graphical output includes visualizing at least a portion of the simulation of the sound propagation in a cut plane of the 3D model in virtual space.

[0386] 25. The wave-based solver of clause 23, wherein the numerical output includes one or more formats such as impulse response, frequency response, energy decay curve, and / or acoustic parameters such as reverberation time, intelligibility, sound pressure level, and / or speech intelligibility.

[0387] 26. The wave-based solver of clause 23, wherein the audible output comprises playing the impulse response convolved with a sound file.

[0388] 27. A computer-implemented method for simulating propagation of sound within at least one model of a virtual domain based on at least one sound source within the virtual domain and at least one acoustic characteristic of the virtual domain, comprising: The computer-implemented method applies a wave-based solver for the step of providing a wave-based simulation by simulating the propagation of the sound using at least a finite element method (FEM), such as a discontinuous Galerkin finite element method (DGFEM) or a spectral element method (SEM).

[0389] 28. The computer-implemented method of clause 27, wherein the virtual domain is a one-dimensional (1D) domain, a two-dimensional (2D) domain, or a three-dimensional (3D) domain such as a three-dimensional model (3D) of a virtual space.

[0390] 29. The computer-implemented method of clause 28, wherein the 3D model of the virtual space defines a bounding surface of an air volume.

[0391] 30. The computer-implemented method of clause 28 or 29, wherein the method represents the boundary surface as a mesh, thereby representing the 3D model of the virtual space as a 3D mesh model.

[0392] 31. The computer-implemented method of clause 30, wherein the 3D mesh model consists of curves.

[0393] 32. The computer-implemented method of any one of clauses 29 to 31, wherein the method represents the air volume as a volumetric 3D mesh model.

[0394] 33. The computer-implemented method of clause 32, wherein the method performs a qualitative assessment on the volumetric 3D mesh model to assess critical areas of sound propagation.

[0395] 34. The computer-implemented method of any one of clauses 27 to 33, wherein the method uses an acoustic wave equation module to numerically discretize the virtual space into subdomains.

[0396] 35. The computer-implemented method of clause 34, wherein the method represents a volumetric 3D mesh model of the air volume as discretized subdomains.

[0397] 36. The computer-implemented method of clause 34 or 35, wherein the method applies the FEM to each subdomain separately.

[0398] 37. A computer-implemented method according to any one of clauses 27 to 36, wherein the method includes simulating the propagation of the sound in at least one first boundary region comprising a first material using locally responsive frequency conditions.

[0399] 38. A computer-implemented method according to any one of clauses 27 to 37, wherein the method includes simulating the propagation of the sound in at least one first boundary region including a second material using an extended response frequency submodule.

[0400] 39. The computer-implemented method of clause 38, wherein the extended reactive frequency dependent boundary modeling submodule is configured to model the second material in a fluid-structure acoustics simulation.

[0401] 40. The computer-implemented method of clause 38 or 39, wherein the second material is a porous material.

[0402] 41. The computer-implemented method of any one of clauses 27 to 40, wherein the method acquires at least one directional sound source in the 3D model, the at least one directional sound source being arranged to emit sound in a specified direction.

[0403] 42. A computer-implemented method according to any one of clauses 27 to 41, wherein the method includes providing a time progression to simulate the propagation of the sound in time and space.

[0404] 43. The computer-implemented method of clause 42, wherein the method includes providing a second time-marching method for simulating the propagation of the sound in time, the first time-marching method comprising a standard low-capacity explicit Runge-Kutta algorithm, and the second time-marching method comprising an implicit-explicit time-marching algorithm.

[0405] 44. The computer-implemented method of any one of clauses 27-43, wherein each subdomain is solved independently on a central processing unit (CPU) and / or a graphics processing unit (GPU).

[0406] 45. The computer-implemented method of any one of clauses 27-44, wherein a message passing interface (MPI) is configured to handle communication between central processing units (CPUs).

[0407] 46. ​​The computer-implemented method of clause 44 or 45, wherein the communication processed by the MPI is performed following a Halo exchange.

[0408] 47. The computer-implemented method of any one of clauses 27 to 46, wherein the method further comprises extracting one or more wave impulse responses based on the wave-based simulation.

[0409] 48. The computer-implemented method of clause 47, wherein the wave impulse response(s) are spatial impulse response(s).

[0410] 49. The computer-implemented method of any one of clauses 27 to 48, wherein the method includes rendering the wave-based simulation for one or more output format(s), such as graphical and / or numerical, and / or audible output format(s).

[0411] 50. The computer-implemented method of clause 49, wherein the step of rendering the graphical output includes visualizing at least a portion of the wave-based simulation in a cross section of the 3D model in virtual space.

[0412] 51. The computer-implemented method of clause 49, wherein the step of rendering the numerical output includes generating one or more formats such as impulse response, frequency response, energy decay curve, and / or acoustic parameters such as reverberation time, intelligibility, sound pressure level, and / or speech intelligibility.

[0413] 52. The computer-implemented method of clause 49, wherein the step of rendering the audible output includes playing the impulse response convolved with a sound file.

[0414] 53. A geometric acoustic solver for simulating the propagation of said sound in at least one model of a virtual domain based on at least one sound source in said virtual domain and at least one acoustic property in said virtual domain, comprising: at least one image source module for determining at least one image source simulation by simulating the propagation of said sound; at least a first acoustic ray tracing module for determining at least a first ray tracing simulation by simulating the propagation of the sound; a hybridization module for combining the at least first ray tracing simulation and the at least one image source simulation into a geometric acoustic simulation of the sound propagation; an output module for preparing an output of the geometric acoustic simulation.

[0415] 54. The geometric acoustic solver of clause 53, wherein the virtual domain is a one-dimensional (1D) domain, a two-dimensional (2D) domain, or a three-dimensional (3D) domain such as a three-dimensional model (3D) of a virtual space.

[0416] 55. The geometric acoustic solver of clause 54, wherein the 3D model of the virtual space defines a bounding surface of an air volume.

[0417] 56. A geometric acoustic solver described in any one of clauses 53 to 55, wherein the at least one image source module determines at least one image source simulation of a first portion of the sound propagation, and the at least first acoustic ray tracing module determines at least a first ray tracing simulation of a second portion of the sound propagation.

[0418] 57. A geometric acoustic solver as described in any one of clauses 53 to 56, wherein the first part of the sound propagation is an initial set of reflections of the sound propagation and the second part of the sound propagation is a subsequent reverberation of the sound propagation.

[0419] 58. A geometric acoustic solver as described in any one of clauses 53 to 57, wherein the initial set of reflections are the first, second, and third reflections, and the subsequent reverberations are the fourth, fifth, and subsequent reflections.

[0420] 59. A geometric acoustic solver described in any one of clauses 53 to 58, wherein the geometric acoustic solver further includes a second acoustic ray tracing module for determining a second ray tracing simulation by simulating the propagation of the sound, and the hybridization module combines the second ray tracing simulation with the first ray tracing simulation and the at least one image source simulation into the geometric acoustic simulation.

[0421] 60. A geometric acoustic solver described in any one of clauses 53 to 59, wherein the first acoustic ray tracing module determines a first sub-part of the first part of the sound propagation and the second acoustic ray tracing module determines a second sub-part of the first part of the sound propagation.

[0422] 61. A geometric acoustic solver described in any one of items 53 to 60, wherein the geometric acoustic solver is further modified for high-performance computing implementation, such as by parallelizing the propagation of each ray or by parallelizing the first and second frequency ranges.

[0423] 62. The geometric acoustic solver of any one of clauses 53 to 61, wherein the geometric acoustic solver further comprises a geometric impulse response module for extracting a geometric impulse response based on the geometric acoustic simulation.

[0424] 63. A geometric acoustic solver according to any one of clauses 53 to 62, wherein the output module prepares the geometric acoustic simulation for one or more output format(s), such as visual and / or graphical and / or numerical, and / or audible output format(s).

[0425] 64. The geometric acoustic solver of clause 63, wherein the graphical output includes visualizing at least a portion of the geometric acoustic simulation on a cut plane of a 3D model of the virtual space.

[0426] 65. The geometric acoustic solver of any one of clauses 53 to 64, wherein the geometric acoustic solver depends on the impedance of the boundary of the air volume.

[0427] 66. The geometric acoustic solver of any one of clauses 53 to 65, wherein the geometric acoustic solver approximates diffraction of acoustic waves in the virtual domain.

[0428] 67. A geometric acoustic solver as described in any one of clauses 53 to 66, wherein the 3D model includes at least one directional sound source for emitting sound in a defined direction.

[0429] 68. A computer-implemented method for simulating the propagation of said sound in at least one model of a virtual domain based on at least one sound source in said virtual domain and at least one acoustic characteristic of said virtual domain, comprising: providing a ray tracing simulation by simulating the propagation of said sound using at least one acoustic ray tracing method; providing an image source simulation by simulating the propagation of said sound using at least one image source method; providing a geometric acoustics simulation of the sound propagation by combining the ray tracing simulation and the image source simulation in a geometric acoustics solver.

[0430] 69. The computer-implemented method of clause 68, wherein the virtual domain is a one-dimensional (1D) domain, a two-dimensional (2D) domain, or a three-dimensional (3D) domain such as a three-dimensional model (3D) of the virtual space.

[0431] 70. The computer-implemented method of clause 69, wherein the 3D model of the virtual space is defined by a bounding surface of an air volume.

[0432] 71. The computer-implemented method of clause 68, wherein the method includes determining the image source simulation of a first portion of the sound propagation, and determining the at least a first ray tracing simulation of a second portion of the sound propagation.

[0433] 72. The computer-implemented method of any one of clauses 68 to 71, wherein the first portion of the sound propagation is the first set of reflections of the sound propagation and the second portion of the sound propagation is the subsequent reverberations of the sound propagation.

[0434] 73. The computer-implemented method of clause 72, wherein the initial set of reflections is a first, second, and third reflection, and the subsequent reverberations are a fourth, fifth, and subsequent reflections.

[0435] 74. The computer-implemented method of any one of clauses 68 to 73, wherein the method further includes providing a second acoustic ray tracing module for determining a second ray tracing simulation by simulating the propagation of the sound, and wherein the hybridization method further includes combining the second ray tracing simulation with the second ray tracing simulation and combining the image source simulation with the geometric acoustic simulation.

[0436] 75. A computer-implemented method according to any one of clauses 68 to 74, wherein the method includes determining a first subpart of the first portion using the first acoustic ray tracing method, and determining a second subpart of the first portion using the second acoustic ray tracing method.

[0437] 76. The computer-implemented method of any one of clauses 68 to 75, wherein the computer-implemented method is further modified for high performance computing implementation, such as by parallelizing the propagation of each ray or by parallelizing the first frequency range and the second frequency range.

[0438] 77. The computer-implemented method of any one of clauses 68 to 76, wherein the method further comprises extracting a geometric spatial impulse response based on the geometric acoustic simulation.

[0439] 78. The computer-implemented method of any one of clauses 68 to 77, wherein the method further comprises preparing the geometric acoustic simulation for one or more output format(s), such as visual and / or graphical and / or numerical and / or audible output format(s).

[0440] 79. The computer-implemented method of clause 78, wherein the graphical output includes visualizing at least a portion of the geometric acoustic simulation in a cross-section of the 3D model of virtual space.

[0441] 80. The computer-implemented method of any one of clauses 68 to 79, wherein the computer-implemented method is configured to depend on the impedance of the boundary of the air volume.

[0442] 81. The computer-implemented method of any one of clauses 68 to 80, wherein the computer-implemented method approximates diffraction of acoustic waves in the virtual domain.

[0443] 82. The computer-implemented method of any one of clauses 68 to 81, wherein the 3D model includes at least one directional sound source for emitting sound in a defined direction.

[0444] 83. A computer-implemented method for acoustic simulation in a virtual domain, comprising: (a) obtaining input data including at least one model of a virtual domain, at least one sound source within said virtual domain, and at least one acoustic characteristic of said virtual domain; (b) obtaining a wave-based solver for determining the wave-based propagation of sound within the virtual domain based on the at least one sound source and the at least one acoustic characteristic within a first acoustic frequency range; (c) obtaining a geometric acoustics-based solver for determining the ray-based propagation of sound within the virtual domain based on the at least one sound source and the at least one acoustic characteristic within a second acoustic frequency range; (d) outputting the wave-based propagation and the geometric acoustic-based propagation.

[0445] 84. The computer-implemented method of clause 83, wherein the virtual domain is a one-dimensional (1D) domain, a two-dimensional (2D) domain, or a three-dimensional (3D) domain such as a three-dimensional model (3D) of a virtual space.

[0446] 85. The computer-implemented method of clause 83, wherein the first acoustic frequency range and the second acoustic frequency range are different.

[0447] 86. The computer-implemented method of clause 83 or 84, wherein the first acoustic frequency range is low-to-mid acoustic frequencies, such as 20 Hz to 2 kHz, and the second acoustic frequency range is high acoustic frequencies, such as 2 kHz to 20 kHz.

[0448] 87. The computer-implemented method of clause 83, wherein the first acoustic frequency range and the second acoustic frequency range partially overlap.

[0449] 88. The computer-implemented method of clause 83, wherein the first acoustic frequency range and the second acoustic frequency range completely overlap.

[0450] 89. The computer-implemented method of any one of clauses 83-88, wherein the wave-based propagation and the geometric acoustic-based propagation are merged together.

[0451] 90. The computer-implemented method of any one of clauses 83-89, wherein the lowest frequency of the second frequency range and / or the highest frequency of the first frequency range are manually selected by a user.

[0452] 91. A computer-implemented method according to any one of clauses 83 to 90, the method comprising a method for applying a wave-based solver according to any one of clauses 26 to 52 and / or a method for applying a geometric acoustic solver according to any one of clauses 68 to 82.

[0453] 92. A method for training a machine learning-driven audio algorithm, comprising: obtaining input data including at least one model of a virtual domain, at least one sound source located within said at least one model of a virtual domain, and at least one acoustic characteristic of said at least one model of a virtual domain; performing at least one sound propagation simulation in a virtual space from the at least one sound source to at least one sound receiver located in the at least one model of a virtual domain; obtaining at least one sound propagation simulation output from the at least one sound propagation simulation; training a machine learning model for a machine learning driven audio algorithm with the at least one sound propagation simulation output.

[0454] 93. The computer-implemented method of clause 92, wherein the virtual domain is a one-dimensional (1D) domain, a two-dimensional (2D) domain, or a three-dimensional (3D) domain such as a three-dimensional model (3D) of a virtual space.

[0455] 94. The computer-implemented method of clause 93, wherein the at least one 3D model includes at least one interior surface, such as a surface of at least one virtual object disposed within the 3D model.

[0456] 95. The computer-implemented method of clause 92, wherein the at least one sound receiver comprises a plurality of oriented receivers.

[0457] 96. The computer-implemented method of clause 95, wherein the plurality of oriented receivers are arranged at the location of the at least one sound receiver, a first arranged receiver being oriented at a first angle and a second arranged receiver being oriented at a second angle.

[0458] 97. The method of any one of clauses 92 to 96, wherein the at least one sound source is at least one directional sound source.

[0459] 98. The method according to any one of clauses 92 to 97, wherein the method includes a step of acquiring at least one raw sound data.

[0460] 99. The method of clause 98, wherein the at least one raw sound data is an audible sound such as music and / or a human voice.

[0461] 100. The method of clause 98 or 99, wherein the at least one raw sound data does not include echo and / or reverberation.

[0462] 101. The method of clause 99, wherein the audible sound is comprised between 20 Hz and 20 kHz.

[0463] 102. The method of any one of clauses 98 to 101, wherein the at least one raw sound data is emitted by the at least one sound source within the 3D model of the virtual space.

[0464] 103. The method of any one of clauses 98 to 102, wherein the at least one sound propagation simulation determines at least one modified raw sound data captured / received by the at least one sound receiver.

[0465] 104. A method according to any one of clauses 98 to 103, wherein the at least one modified raw sound data is the at least one raw sound data convolved with a simulated impulse response obtained by the at least one sound propagation simulation.

[0466] 105. A method according to any one of clauses 98 to 104, wherein the machine learning model is trained by comparing the at least one raw sound data with the at least one modified raw sound data.

[0467] 106. A method according to any one of clauses 98 to 105, wherein the machine learning model is trained by convolving the at least one raw sound data with a simulated impulse response obtained by the at least one sound propagation simulation.

[0468] 107. The method of any one of clauses 98 to 106, wherein the input data includes a plurality of three-dimensional models of the virtual space, preferably three or more three-dimensional models of the virtual space, more preferably ten or more three-dimensional models of the virtual space.

[0469] 108. The method of any one of clauses 98-107, wherein the at least one sound source is substantially co-located with the at least one sound receiver.

[0470] 109. A method according to any one of clauses 98 to 108, wherein the at least one sound propagation simulation has at least one simulation time.

[0471] 110. The method of clause 109, wherein at least one simulation time is calibrated according to the 3D model(s) characteristics.

[0472] 111. The method of clause 110, wherein the 3D model(s) include the 3D model(s) geometry and / or size.

[0473] 112. The method of any one of clauses 92 to 111, wherein the method is used for speech recognition, echo cancellation, blind source separation, blind room response modeling, or feature extraction.

[0474] 113. The method of any one of clauses 92 to 112, wherein the at least one sound propagation simulation is configured to use the solver(s) and / or the computer-implemented method(s) according to clauses 1 to 91.

[0475] 114. A method for determining a head-related transfer function, comprising: (a) obtaining the geometry of the user's head; (b) performing a simulation of sound propagation from at least one audio source to the geometry of the user's head, wherein the simulation of sound propagation is based on a sound propagation simulation using a wave-based solver; (c) determining a head-related transfer function (HRTF) of the user's head based on the simulation of sound propagation.

[0476] 115. The method of clause 114, wherein the geometry of the user's head includes the geometry of at least one ear.

[0477] 116. The method of any one of clauses 114 to 115, wherein the wave-based solver is a discontinuous Galerkin finite element method-based solver.

[0478] 117. A method according to any one of clauses 114 to 116, wherein the wave-based solver determines the simulation of sound propagation in an acoustic frequency range.

[0479] 118. The method of clause 117, wherein the acoustic frequency range comprises 20 Hz to 20 kHz.

[0480] 119. A method according to any one of clauses 114 to 118, wherein the wave-based solver determines a simulation of the sound propagation within a first acoustic frequency range.

[0481] 120. The method of clause 119, wherein the first acoustic frequency range comprises 20 Hz to 200 Hz.

[0482] 121. A method according to any one of clauses 114 to 120, wherein the wave-based solver determines a simulation of the sound propagation within a second acoustic frequency range.

[0483] 122. The method of clause 121, wherein the second acoustic frequency range comprises 20 Hz to 200 Hz.

[0484] 123. A method according to any one of clauses 114 to 122, wherein the wave-based solver determines a simulation of the sound propagation within a third acoustic frequency range.

[0485] 124. The method of clause 123, wherein the third acoustic frequency range is comprised between 20 Hz and 20 kHz, preferably between 20 Hz and 6 kHz, more preferably between 20 Hz and 10 kHz.

[0486] 125. A method according to any one of clauses 114 to 124, wherein the simulation of sound propagation using a wave-based solver is configured to use a wave-based solver according to any one of clauses 1 to 26 and / or a computer-implemented method according to any one of clauses 27 to 52.

[0487] 126. A computer-implemented method for generating a binaural auralization at a listening point, comprising: (a) determining a head-related transfer function (HRTF) of a user's head in accordance with any of clauses 114 to 125; (b) receiving a 3D model of a domain, a location of at least one sound source within the domain, and acoustic properties of a plurality of boundaries within the 3D model of the domain; (c) arranging an array of secondary receivers around the listening point; (d) executing a wave-based solver to determine a wave impulse response of the wave-based propagation of an impulse emitted by at least one sound source within the 3D model of the domain and received at the listening point within a first acoustic frequency range; (e) determining wave density coefficients based on predictions for spherical harmonic basis functions at the listening point; (f) generating a binaural auralization at the listening point by evaluating a spatial rendering equation, the spatial rendering equation combining the wave density coefficients and the HRTFs.

[0488] 127. The computer-implemented method of clause 126, wherein the method further comprises convolving the binaural auralization with echo-free sound, thereby generating sound rendered at the listening point.

[0489] 128. The computer-implemented method of any one of clauses 126 or 127, wherein the array of secondary receivers is an open spherical array.

[0490] 129. The computer-implemented method of clause 128, wherein the secondary receivers are arranged in the open spherical array.

[0491] 130. The computer-implemented method of any one of clauses 126-129, wherein the secondary receiver is a directional receiver.

[0492] 131. The computer-implemented method of clause 130, wherein the directional receiver is a directional receiver having a first-order cardioid directional pattern.

[0493] 132. The computer-implemented method of any one of clauses 126 to 131, wherein the method further comprises executing a geometric acoustics-based solver to determine a geometric acoustics impulse response of a ray-based propagation of the impulse emitted by the at least one sound source within the 3D model of the domain and received at the listening point within a second acoustic frequency range.

[0494] 133. The computer-implemented method of clause 132, wherein the method further comprises generating the impulse response by combining the wave impulse response and the geometric impulse response.

[0495] 134. The computer-implemented method of clause 132 or 133, wherein the method further comprises determining geometric acoustic spherical harmonic coefficients from the at least one sound source to the listening point.

[0496] 135. The computer-implemented method of clause 132, 133, or 134, wherein the method further comprises combining the geometric acoustic spherical harmonic coefficients and the wave density coefficients, thereby obtaining hybridized spherical harmonic coefficients in the first frequency range and the second frequency range.

[0497] 136. The computer-implemented method of any one of clauses 126 to 135, wherein the method further comprises rotating an HRTF reference to establish a given orientation at the listening point.

[0498] 137. The computer-implemented method of any one of clauses 126 to 136, wherein the first frequency range and the second frequency range are the first frequency range and the second frequency range of any one of clauses 114 to 125.

[0499] 138. A computer-implemented method for generating an impulse response for a listening point in a room, comprising: (a) receiving a 3D model of the room, the location of at least one sound source within the 3D model of the room, and acoustic characteristics of the boundaries within the 3D model of the room; (b) executing a wave-based solver to determine a wave impulse response of the wave-based propagation of impulses emitted by the at least one sound source within the 3D model of the room and received at the listening point within a first acoustic frequency range; (c) executing a geometric acoustics-based solver to determine a geometric impulse response of a ray-based propagation of an impulse emitted by the at least one sound source within the 3D model of the room and received at the listening point within a second acoustic frequency range; (d) generating the impulse response by combining the wave impulse response and the geometric impulse response.

[0500] 139. A computer-implemented method for generating a binaural auralization at a listening point, comprising: (a) receiving a 3D model of the room, a location of at least one sound source within the 3D model of the room, and acoustic characteristics of at least one boundary within the 3D model of the room; (b) using a wave-based solver to determine a wave-based impulse response of a wave-based propagation of an impulse emitted by the at least one sound source within the 3D model of the room and received at the listening point within a first acoustic frequency range; (c) using a geometric acoustics-based solver to determine a geometric impulse response of a ray-based propagation of an impulse emitted by the at least one sound source within the 3D model of the room and received at the listening point within a second acoustic frequency range; (d) generating the impulse response by merging the wave impulse response and the geometric impulse response.

[0501] 140. The computer-implemented method of clause 139, wherein the first acoustic frequency range and the second acoustic frequency range partially overlap.

[0502] 141. The computer-implemented method of clause 139 or 140, wherein the virtual domain includes at least one directional sound source for emitting sound in a defined direction.

[0503] 142. The computer-implemented method of any one of clauses 139 to 141, wherein the computer-implemented method further comprises executing a mesh model of the 3D model of the room, the 3D mesh model being a 3D curved mesh model.

[0504] 143. The computer-implemented method of any one of clauses 139-142, wherein the wave-based solver applies a discontinuous Galerkin finite element method (DGFEM) or a spectral element method (SEM).

[0505] 144. The computer-implemented method of any one of clauses 139 to 143, wherein the method further comprises a calibration step in which the power level of the at least one sound source is adjusted so that the sound level received at a predetermined distance from the at least one sound source is the same for the wave-based solver and the geometric acoustic solver.

[0506] 145. The computer-implemented method of any one of clauses 139-144, wherein the upper frequency of the first acoustic frequency range and the lower frequency of the second acoustic frequency range overlap at a transition frequency.

[0507] 146. The computer-implemented method of clause 145, wherein merging the wave-based impulse response and the geometric impulse response includes applying a low-pass filter to the wave-based impulse response.

[0508] 147. The computer-implemented method of clause 145 or 146, wherein merging the wave-based impulse response and the geometric impulse response includes applying a high-pass filter to the geometric impulse response.

[0509] 148. The computer-implemented method of clause 146 or 147, wherein the low-pass filter and / or the high-pass filter include a cutoff frequency at the transition frequency.

[0510] 149. The computer-implemented method of any one of clauses 139 to 148, wherein the wave-based solver and / or the geometric acoustic solver comprises extracting one or more wave-based impulse response(s) and / or one or more geometric impulse response(s) based on the simulation of the sound propagation, and the wave-based impulse response(s) are spatial impulse response(s).

[0511] 150. The computer-implemented method of clause 149, wherein the spatial impulse response(s) include a plurality of single channel impulse responses, each of the plurality of single channel impulse responses recording the wave impulse response from a particular direction or angle at the same listening point.

[0512] 151. The computer-implemented method of any one of clauses 139-150, wherein a spherical receiver array is arranged around the listening point, the spherical receiver array including a plurality of receivers.

[0513] 152. The computer-implemented method of clause 151, wherein the spherical receiver array is an open spherical array of cardioid receivers.

[0514] 153. The computer-implemented method of clause 151 or 152, wherein the spherical receiver array includes at least two receivers, preferably at least four receivers, more preferably at least eight receivers, even more preferably at least 16 receivers, most preferably at least 32 receivers, and even most preferably at least 64 receivers.

[0515] 154. The number of receivers is determined based on the maximum truncation order N, so that the number of receivers is (N+1) 2 The computer-implemented method of any one of clauses 151 to 153, wherein the above is determined.

[0516] 155. The computer-implemented method of any one of clauses 139 to 154, wherein the computer-implemented method further comprises convolving the generated impulse response with a basis audio signal so as to generate a convolved audio signal.

[0517] 156. The computer-implemented method of any one of clauses 139 to 155, wherein the computer-implemented method further includes rendering the basis audio signal by convolving the basis audio signal with the generated impulse response, thereby generating a rendered audio signal.

[0518] 157. The computer-implemented method of clause 156, wherein the underlying audio signal is speech, music, environmental sound, impulse sound, or any combination thereof.

[0519] 158. The computer-implemented method of any one of clauses 155 to 157, wherein the rendered audio signal provides an audio rendering of the base audio signal in the 3D model of the room at the listening point.

[0520] 159. A system for generating an impulse response to a listening point in a room, comprising: A computer system having a processor coupled to a memory, the processor comprising: receiving a 3D model of the room, a location of at least one sound source within the 3D model of the room, and acoustic characteristics of at least one boundary within the 3D model of the room; determining a wave-based impulse response for wave-based propagation of an impulse emitted by the at least one sound source within the 3D model of the room and received at the listening point within a first acoustic frequency range using a wave-based solver; determining a geometric impulse response of a ray-based propagation of an impulse emitted by the at least one sound source within the 3D model of the room and received at the listening point within a second acoustic frequency range using a geometric acoustic-based solver; The system is configured to generate the impulse response by merging the wave impulse response and the geometric impulse response.

[0521] 160. A computer-implemented method for training a machine learning model for audio compensation, comprising: - receiving 3D models of a plurality of rooms, each of said 3D models including at least one sound source and at least one acoustic property; - receiving a plurality of impulse responses at listening positions in each of the plurality of rooms; - training a machine learning model for the audio compensation using at least the plurality of impulse responses as input.

[0522] 161. The computer-implemented method of clause 160, wherein the plurality of impulse responses are pre-processed to generate a plurality of modified impulse responses for training the machine learning model.

[0523] 162. The computer-implemented method of clause 161, wherein the pre-processing includes applying a filter to emphasize a voiced range in the impulse response.

[0524] 163. The computer-implemented method of clause 162, wherein the audio range is 3 kHz to 17 kHz, or 350 Hz to 17 kHz.

[0525] 164. The computer-implemented method of any one of clauses 160-163, wherein the method further includes using a plurality of reverberant audio signals as inputs for training the machine learning model to provide the plurality of reverberant audio signals.

[0526] 165. The computer-implemented method of clause 164, wherein the reverberant audio signal is provided by convolving each of the plurality of impulse responses with a basis audio signal.

[0527] 166. The computer-implemented method of clause 164, wherein the reverberant audio signal is provided by recording an audio signal received by at least one speaker of an audio device.

[0528] 167. The computer-implemented method of any one of clauses 160-166, wherein the method further comprises using the 3D models of the plurality of rooms as input for training the machine learning model.

[0529] 168. The computer-implemented method of any one of clauses 160-167, wherein the method further includes receiving a digital model of the audio device and using the digital model of the audio device as input for training the machine learning model.

[0530] 169. The computer-implemented method of any one of clauses 160-168, wherein at least one preferred listening position for each of the plurality of rooms is used as input for training the machine learning model.

[0531] 170. The computer-implemented method of any one of clauses 160-169, wherein training the machine learning model includes using a plurality of reverberant audio signals described in clauses 164, 165, or 166 as inputs, and wherein training includes reestablishing basis audio signals as outputs.

[0532] 171. The computer-implemented method of clause 170, wherein the 3D models of the plurality of rooms are used as input and the at least one preferred listening point for each of the plurality of rooms is used as input, the method further comprising re-establishing the base audio signal as output at the at least one preferred listening point.

[0533] 172. The computer-implemented method of any one of clauses 160-171, wherein training the machine learning model includes generating a compensation impulse response as an output.

[0534] 173. The computer-implemented method of clause 172, wherein generating the compensation impulse response is based on the reverberant audio signal of clause 164, 165, or 166 and basis audio signals as inputs for training the machine learning model.

[0535] 174. The computer-implemented method of clause 172 or 173, wherein the 3D models of the plurality of rooms are used as input, and the at least one preferred listening point for each of the plurality of rooms is used as input, and the method further comprises generating the compensation impulse response at the at least one preferred listening point as output.

[0536] 175. The computer-implemented method of any one of clauses 160 to 174, wherein the method further comprises: receiving the plurality of impulse responses comprises generating the impulse responses for each 3D model of the plurality of rooms according to the method described in clauses 139 to 158.

[0537] 176. The computer-implemented method of any one of clauses 160-175, wherein the method further comprises receiving a digital model of the audio device.

[0538] 177. The computer-implemented method of any one of clauses 160-176, wherein training the machine learning model includes a neural network.

[0539] 178. The computer-implemented method of clause 177, wherein the neural network includes an autoencoder for encoding any of the inputs to the model or for generating compressed inputs for use in training the machine learning model for audio compensation.

[0540] 179. The computer-implemented method of clause 177 or 178, wherein the neural network further includes training a generative adversarial network (GAN) to generate any one of the inputs for training a machine learning model for audio compensation.

[0541] 180. The computer-implemented method of clause 177, 178, or 179, wherein the neural network includes a deep neural network, a convolutional neural network, and / or a transformer for training a machine learning model for audio compensation.

[0542] 181. A method for providing a machine learning model for audio compensation in an audio device, the audio device including at least one microphone, and the machine learning model being trained by a method according to any one of clauses 160 to 180, the method comprising: receiving an audio signal at the at least one microphone; and generating a compensated audio signal by using the machine learning model for audio compensation.

[0543] 182. A method for providing machine learning based audio compensation in an audio device including at least one microphone, wherein the machine learning based audio compensation has been trained using a computer-implemented method for training a machine learning model described in any one of clauses 160 to 180, the method comprising: receiving an audio signal at the at least one microphone; and generating a compensated audio signal by using the trained machine learning model on the received audio signal.

[0544] 183. The method of clause 181 or 182, wherein the compensated audio signal is compensated by convolving the received audio signal with the compensation impulse response of clause 172, 173 or 174.

[0545] 184. An audio device including at least one microphone, the audio device including a processing system for applying machine learning based audio compensation according to any one of clauses 181 to 183, the audio device configured to receive an audio signal at the at least one microphone and apply the machine learning based audio compensation to the audio signal to generate a compensated audio signal.

[0546] 185. The audio device of clause 184, wherein the audio device includes a communications module configured to transmit the compensated audio signal.

[0547] 186. The audio device of clause 184 or 185, wherein the audio device is configured to transmit the compensated audio signal to the cloud, the Internet, and / or a network storage center.

[0548] 187. The audio device of clause 184, 185 or 186, wherein the audio device is further configured to transmit the compensated audio signal to a remote audio device, the remote audio device including at least one remote speaker for outputting the compensated audio signal.

[0549] 188. A system for training a machine learning model for audio compensation, the system comprising: A computer system having a processor coupled to a memory, the processor comprising: receiving 3D models of a plurality of rooms, each of the 3D models including at least one sound source and at least one acoustic characteristic; receiving a plurality of impulse responses at listening positions in each of the plurality of rooms; The system is configured to train a machine learning model for audio compensation using at least the plurality of impulse responses as inputs.

[0550] 189. A computer-implemented method for determining a head-related transfer function, comprising: (a) receiving a 3D model of a user's head and a location of at least one sound source within the 3D model representing an eardrum or a vicinity thereof; (b) using a wave-based solver for determining a plurality of wave-based impulse responses from impulses emitted by the at least one sound source, the plurality of wave-based impulse responses being determined from a plurality of digital representations of head-mounted receivers; (c) determining a head-related transfer function (HRTF) of the user's head based on determining the plurality of wave-based impulse responses at a plurality of digital representations of the head-receivers.

[0551] 190. The computer-implemented method of clause 189, wherein the wave-based solver uses a finite element method (FEM), such as a discontinuous Galerkin finite element method (DGFEM) or a spectral element method (SEM).

[0552] 191. The computer-implemented method of any one of clauses 189-190, wherein the computer-implemented method further comprises obtaining a head mesh model representing the geometry of the user's head.

[0553] 192. The computer-implemented method of clause 191, wherein the head mesh model is a curvilinear head mesh model.

[0554] 193. The computer-implemented method of any one of clauses 189 to 192, wherein the computer-implemented method further includes arranging a digital representation of a head array comprising a plurality of digital representations of the head receivers around the head mesh model, such that a distance between any of the digital representations of the head receivers and the head mesh model is less than a predetermined distance.

[0555] 194. The computer-implemented method of any one of clauses 189 to 193, wherein the computer-implemented method further includes determining a first closest mesh element on the head mesh model that is closest to the eardrum.

[0556] 195. The computer-implemented method of any one of clauses 189 to 194, wherein the computer-implemented method further includes arranging the digital representation of a first source-compensated microphone located at a first source distance from a first nearest mesh element, the first source distance being less than the predetermined distance.

[0557] 196. The computer-implemented method of any one of clauses 189-195, wherein the computer-implemented method further comprises digitally emitting a first impulse signal using the first nearest mesh element as a sound source.

[0558] 197. The computer-implemented method of any one of clauses 189 to 196, wherein the computer-implemented method further includes determining a first source correction signal using the wave-based solver, the first source correction signal describing the first impulse signal received at the first source correction microphone.

[0559] 198. The computer-implemented method of any one of clauses 189 to 197, wherein the computer-implemented method further includes determining a plurality of first source-corrected head impulse responses by source-correcting each of the plurality of wave-based impulse responses using the first source-corrected signal.

[0560] 199. The computer-implemented method of any one of clauses 189 to 198, further comprising generating the head-related transfer function of the user's head relative to the first eardrum by combining the corrected head-related impulse responses of the plurality of first sources.

Claims

1. 1. A computer-implemented method for generating an impulse response for a listening point in a room, comprising: (a) receiving a 3D model of the room, a location of at least one sound source in the 3D model of the room, and acoustic characteristics of at least one boundary in the 3D model of the room; (b) using a wave-based solver to determine a wave-based impulse response of a wave-based propagation of an impulse emitted by the at least one sound source within the 3D model of the room and received at the listening point within a first acoustic frequency range; (c) using a geometric acoustics-based solver to determine a geometric impulse response of a ray-based propagation of an impulse emitted by the at least one sound source within the 3D model of the room and received at the listening point within a second acoustic frequency range; (d) generating the impulse response by merging the wave impulse response and the geometric impulse response.

2. The computer-implemented method of claim 1 , wherein the first and second acoustic frequency ranges partially overlap.

3. The computer-implemented method of claim 1 or 2, wherein the virtual domain includes at least one directional sound source for emitting sound in a defined direction.

4. 10. The computer-implemented method of any one of the preceding claims, further comprising: executing a mesh model of the 3D model of the room, wherein the 3D mesh model is a 3D curvilinear mesh model.

5. The computer-implemented method of claim 1 , 2 or 3 , wherein the wave-based solver applies a discontinuous Galerkin finite element method (DGFEM) or a spectral element method (SEM).

6. 6. The computer-implemented method of claim 1, wherein the method further comprises a calibration step, wherein a power level of the at least one sound source is adjusted so that a sound level received at a predetermined distance from the at least one sound source is the same for the wave-based solver and the geometric acoustic solver.

7. The computer-implemented method of any one of claims 1 to 6, wherein an upper frequency limit of the first acoustic frequency range and a lower frequency limit of the second acoustic frequency range overlap at a transition frequency.

8. The computer-implemented method of claim 7 , wherein merging the wave-based impulse response and the geometric impulse response comprises applying a low-pass filter to the wave-based impulse response.

9. The computer-implemented method of claim 7 or 8, wherein merging the wave-based impulse response and the geometric impulse response comprises applying a high-pass filter to the geometric impulse response.

10. 10. A computer-implemented method according to claim 8 or 9, wherein the low-pass filter and / or the high-pass filter comprises a cut-off frequency at the transition frequency.

11. 10. The computer-implemented method of claim 1, wherein the wave-based solver and / or the geometric acoustic solver comprises extracting one or more wave-based impulse response(s) and / or one or more geometric impulse response(s) based on the simulation of the sound propagation, wherein the wave-based impulse response(s) are spatial impulse response(s).

12. 12. The computer-implemented method of claim 11, wherein the spatial impulse response(s) include a plurality of single-channel impulse responses, each of the plurality of single-channel impulse responses recording the wave impulse response from a particular direction or angle at the same listening point.

13. 10. A computer-implemented method according to any one of the preceding claims, wherein a spherical receiver array is arranged around the listening point, the spherical receiver array comprising a plurality of receivers.

14. 14. The computer-implemented method of claim 13, wherein the spherical receiver array is an open spherical array of cardioid receivers.

15. 15. The computer-implemented method of claim 13 or 14, wherein the spherical receiver array comprises at least two receivers, preferably at least four receivers, more preferably at least eight receivers, even more preferably at least 16 receivers, most preferably at least 32 receivers, and even most preferably at least 64 receivers.

16. The number of receivers is determined based on the maximum truncation order N, such that the number of receivers is (N+1) ２ The computer-implemented method of any one of claims 13 to 15, wherein the above is determined.

17. 10. A computer-implemented method according to any one of the preceding claims, further comprising convolving the generated impulse responses with basis audio signals so as to generate a convolved audio signal.

18. 10. A computer-implemented method according to any one of the preceding claims, wherein the computer-implemented method further comprises rendering a basis audio signal by convolving the basis audio signal with the generated impulse response, thereby generating a rendered audio signal.

19. 20. The computer-implemented method of claim 18, wherein the underlying audio signal is speech, music, environmental sound, impulse sound, or any combination thereof.

20. 20. The computer-implemented method of any one of claims 17 to 19, wherein the rendered audio signal provides an audio rendering of the underlying audio signal in the 3D model of the room at the listening point.

21. 1. A system for generating an impulse response for a listening point in a room, comprising: A computer system having a processor coupled to a memory, the processor comprising: receiving a 3D model of the room, a location of at least one sound source within the 3D model of the room, and acoustic characteristics of at least one boundary within the 3D model of the room; determining, using a wave-based solver, a wave-based impulse response of a wave-based propagation of an impulse emitted by the at least one sound source within the 3D model of the room and received at the listening point within a first acoustic frequency range; determining, using a geometric acoustics-based solver, a geometric impulse response of ray-based propagation of impulses emitted by the at least one sound source within the 3D model of the room and received at the listening point within a second acoustic frequency range; The system is configured to generate the impulse response by merging the wave impulse response and the geometric impulse response.

22. 1. A computer-implemented method for training a machine learning model for audio compensation, comprising: - receiving 3D models of a plurality of rooms, each of said 3D models including at least one sound source and at least one acoustic property; - receiving a plurality of impulse responses at listening positions in each of said plurality of rooms; - training a machine learning model for said audio compensation using at least said plurality of impulse responses as input; The computer-implemented method includes:

23. 23. The computer-implemented method of claim 22, wherein the plurality of impulse responses are pre-processed to generate a plurality of modified impulse responses for training the machine learning model.

24. 24. The computer-implemented method of claim 23, wherein the pre-processing includes applying a filter to increase audio range in the impulse response.

25. 25. The computer-implemented method of claim 24, wherein the audio range is between 3 kHz and 17 kHz or between 350 kHz and 17 kHz.

26. 26. The computer-implemented method of claim 22, further comprising using a plurality of reverberant audio signals as inputs for training the machine learning model to provide the plurality of reverberant audio signals.

27. 27. The computer-implemented method of claim 26, wherein the reverberant audio signal is provided by convolving each of the plurality of impulse responses with a basis audio signal.

28. 27. The computer-implemented method of claim 26, wherein the reverberant audio signal is provided by recording an audio signal received by at least one speaker of an audio device.

29. 29. The computer-implemented method of any one of claims 22 to 28, wherein the method further comprises using the 3D models of the plurality of rooms as input for training the machine learning model.

30. 30. The computer-implemented method of any one of claims 22 to 29, wherein the method further comprises receiving a digital model of the audio device and using the digital model of the audio device as an input for training the machine learning model.

31. 31. The computer-implemented method of any one of claims 22 to 30, wherein at least one preferred listening position for each of the plurality of rooms is used as input for training the machine learning model.

32. 32. The computer-implemented method of any one of claims 22 to 31, wherein training the machine learning model comprises using a plurality of reverberant audio signals according to claim 26, 27, or 28, and wherein training comprises re-establishing basis audio signals as outputs.

33. 33. The computer-implemented method of claim 32, wherein the 3D models of the plurality of rooms are used as inputs and the at least one preferred listening point in each of the plurality of rooms is used as inputs, the method further comprising re-establishing the base audio signal at the at least one preferred listening point as an output.

34. 34. The computer-implemented method of any one of claims 22 to 33, wherein training the machine learning model includes producing a compensation impulse range as an output.

35. 35. The computer-implemented method of claim 34, wherein the compensation impulse response is based on the reverberant audio signal of claim 26, 27, or 28 and basis audio signals as inputs for training the machine learning model.

36. 36. The computer-implemented method of claim 34 or 35, wherein 3D models of the plurality of rooms are used as input and the at least one preferred listening point in each of the plurality of rooms is used as input, the method further comprising generating the compensation impulse response as output at the at least one preferred listening point.

37. 37. The computer-implemented method of any one of claims 22 to 36, wherein receiving the plurality of impulse responses comprises generating the impulse responses for a 3D model of each of the plurality of rooms according to a method of claims 1 to 21.

38. The computer-implemented method of any one of claims 22 to 37, wherein the method further comprises receiving a digital model of the audio device.

39. 39. The computer-implemented method of any one of claims 22 to 38, wherein training the machine learning model comprises a neural network.

40. 40. The computer-implemented method of claim 39, wherein the neural network includes an autoencoder for encoding any of the inputs to the model or for generating compressed inputs for use in training a machine learning model for the audio compensation.

41. 41. The computer-implemented method of claim 39 or 40, wherein the neural network further comprises training a generative adversarial network (GAN) to generate any one of the inputs for training the machine learning model for audio compensation.

42. 42. The computer-implemented method of claim 39, 40, or 41, wherein the neural network comprises a deep neural network, a convolutional neural network, and / or a transformer for training a machine learning model for the audio compensation.

43. 43. A method for providing a machine learning model for audio compensation in an audio device, the audio device including at least one microphone, the machine learning model being trained according to the method of any one of claims 22 to 42, the method comprising: receiving an audio signal with the at least one microphone; and generating a compensated audio signal by using the machine learning model for audio compensation.

44. 43. A method for providing machine learning-based audio compensation in an audio device including at least one microphone, wherein the machine learning-based audio compensation has been trained using a computer-implemented method for training a machine learning model as described in any one of claims 22 to 42, the method comprising: receiving an audio signal with the at least one microphone; and generating a compensated audio signal by applying the trained machine learning model to the received audio signal.

45. 45. The method of claim 43 or 44, wherein the compensated audio signal is compensated by convolving the received audio signal with the compensation impulse response of claim 34, 35 or 36.

46. 46. ​​An audio device including at least one microphone, the audio device comprising a processing system for applying machine learning based audio compensation according to a method of any one of claims 43 to 45, the audio device being configured to receive an audio signal at the at least one microphone and to apply the machine learning based audio compensation to the audio signal to generate a compensated audio signal.

47. 47. The audio device of claim 46, wherein the audio device includes a communications module configured to transmit the compensated audio signal.

48. 48. The audio device of claim 46 or 47, wherein the audio device is configured to transmit the compensated audio signal to the cloud, the internet, and / or a network storage center.

49. 49. The audio device of claim 46, 47, or 48, wherein the audio device is further configured to transmit the compensated audio signal to a remote audio device, the remote audio device including at least one remote speaker for outputting the compensated audio signal.

50. 1. A system for training a machine learning model for audio compensation, the system comprising: A computer system having a processor coupled to a memory, the processor comprising: receiving 3D models of a plurality of rooms, each of the 3D models including at least one sound source and at least one acoustic characteristic; receiving a plurality of impulse responses at listening positions in each of the plurality of rooms; The system is configured to train a machine learning model for audio compensation using at least the plurality of impulse responses as inputs.

51. 1. A computer-implemented method for determining a head-related transfer function, comprising: (a) receiving a 3D model of a user's head and a location of at least one sound source within the 3D model representing an eardrum or a vicinity thereof; (b) using a wave-based solver for determining a plurality of wave-based impulse responses from impulses emitted by the at least one sound source, the plurality of wave-based impulse responses being determined from a plurality of digital representations of head-mounted receivers; (c) determining a head-related transfer function (HRTF) of the user's head based on the wave-based impulse responses determined at the digital representations of the head-receivers; The method comprising:

52. 52. The computer-implemented method of claim 51, wherein the wave-based solver uses a discontinuous Galerkin finite element method (DGFEM) or a spectral element method (SEM).

53. The computer-implemented method of any one of claims 51 to 52, further comprising obtaining a head mesh model representing a geometry of the user's head.

54. 54. The computer-implemented method of claim 53, wherein the head mesh model is a curved head mesh model.

55. 55. The computer-implemented method of any one of claims 51 to 54, further comprising arranging a digital representation of a head array comprising a plurality of digital representations of the head receivers around the head mesh model, such that a distance between any of the digital representations of the head receivers and the head mesh model is less than a predetermined distance.

56. 56. The computer-implemented method of any one of claims 51 to 55, further comprising determining a first closest mesh element on the head mesh model that is closest to the eardrum.

57. 57. The computer-implemented method of claim 51, further comprising arranging the digital representation of a first source-compensated microphone located at a first source distance from the first nearest mesh element, the first source distance being less than a predetermined distance.

58. 58. The computer-implemented method of any one of claims 51 to 57, further comprising digitally emitting a first impulse signal using the first nearest mesh element as an acoustic source.

59. 59. The computer-implemented method of any one of claims 51 to 58, further comprising using the wave-based solver to determine a first source correction signal, the first source correction signal describing the first impulse signal received at the first source correction microphone.

60. 60. The computer-implemented method of any one of claims 51 to 59, further comprising determining a plurality of first source-corrected head-based impulse responses by source-correcting each of the plurality of wave-based impulse responses using the first source correction signal.

61. 61. The computer-implemented method of claim 51, further comprising generating the head-related transfer function of the user's head relative to the first eardrum by combining corrected head-related impulse responses of the plurality of first sources.

Images