Method and system for optimizing immersive directional sound simulation based on virtual engine

Abstract

The application relates to the technical field of virtual engine and discloses an immersive directional sound effect simulation optimization method and system based on a virtual engine. The method submits an acoustic ray tracing task to a hardware light tracing unit of a graphic processor at an acoustic solving rate lower than a graphic frame rate, accesses the same level of a bounding box acceleration structure as a graphic light tracing task, solves an effective sound propagation path according to a graphic material identification of a hit triangular facet, generates a path signature according to a sound source and the hit triangular facets successively passed through, and packs the path signature into an acoustic state snapshot for output; an audio rendering thread runs at an audio block rate higher than the acoustic solving rate, matches the snapshot and an active path set according to the signature, distinguishes a continuous path, a newborn path and a dying path, and respectively performs smoothing, fading in and fading out, and performs binauralization rendering on a sound source signal to obtain output binaural signals. The application solves the acoustic by reusing the graphic side acceleration structure, and continuously presents the directional sound effect without artifacts at a lower solving rate.

Description

Technical Field

[0001] This invention belongs to the field of virtual engine technology, specifically relating to an immersive directional sound effect simulation optimization method and system based on a virtual engine. Background Technology

[0002] With the development of applications such as virtual reality, video games, and digital twins, users are demanding increasingly higher levels of spatial realism in sound within virtual scenes. The sense of direction and distance of sound, as well as reflections and reverberation that occur with scene changes, are crucial for creating immersion. Existing spatial sound effects approaches can be broadly categorized into two types. The first is the pre-computation approach, which solves the sound field offline during the scene construction phase, baking propagation parameters into lookup tables, and then superimposing these tables at runtime. This approach has lower runtime overhead; however, the baking results are tied to scene geometry. Once objects in the scene move, open, close, or deform, the pre-computation data becomes ineffective, making it difficult to adapt to the numerous dynamic geometries present in virtual engines. The second type is the runtime geometric acoustic approach, which solves reflections online using ray tracing or mirror source methods. This approach can follow scene changes; however, acoustic solutions require repeated intersection calculations of rays and triangular faces based on scene geometry. The computational load increases rapidly with scene complexity and reflection order, often requiring separate maintenance of geometric representations and intersection structures for acoustics, consuming considerable memory bandwidth and processor resources, and competing with graphics rendering for computational power. Furthermore, acoustic solutions typically operate at lower frequencies, while auditory signals require continuous output at higher sampling rates. This difference in rates causes abrupt changes in the time delay, direction, and intensity of the reflection path between adjacent solution results, manifesting as pops, directional jitter, and abrupt changes in timbre. Current technology lacks a spatial sound effect implementation method that can both follow dynamic scenes and suppress artifacts caused by the difference between solution and output rates. Summary of the Invention

[0003] The main objective of this invention is to provide an immersive directional sound effect simulation optimization method and system based on a virtual engine, in order to solve the problems in the prior art where acoustic solution requires maintaining a separate geometric representation for sound, resulting in high overhead, and the gap between acoustic solution rate and audio output rate causing pops, directional jitter, and abrupt changes in timbre in directional sound effects. While following the dynamic scene, it continuously presents spatial sound effects with a sense of direction and distance without artifacts at a low acoustic solution rate.

[0004] To solve the above problems, the technical solution of the present invention is implemented as follows:

[0005] An immersive directional sound effect simulation optimization method based on a virtual engine is applied to a virtual engine running a graphics rendering pipeline. The graphics rendering pipeline constructs or incrementally updates the hierarchical bounding box acceleration structure of scene geometry in each graphics frame, including the following steps:

[0006] Step 1: Periodically perform acoustic propagation solving at an acoustic solution rate lower than the graphics frame rate and set independently. Submit the acoustic ray tracing task to the hardware ray tracing unit of the graphics processor. The acoustic ray tracing task and the graphics ray tracing task access the same bounding box acceleration structure. Determine the acoustic parameters in a predetermined number of frequency bands based on the graphics material identifier carried by the hit triangle facets and use them to determine the frequency band energy gain. Solve to obtain the effective sound propagation path from the sound source to the listener. The path parameters of each effective sound propagation path include propagation delay, direction of arrival, and frequency band energy gain. Generate a path signature for each effective sound propagation path based on the sound source and the hit triangle facets passed through in sequence. Pack the path parameters and path signatures of all effective sound propagation paths into an acoustic state snapshot for output.

[0007] Step 2: The audio rendering thread runs continuously in units of audio blocks and maintains the active path set. The audio block rate is higher than the acoustic solution rate. Whenever a new acoustic state snapshot is adopted, the path signature set of the acoustic state snapshot is matched with the path signature set of the active path set: paths whose path signatures exist in both sets are persistent paths; paths whose path signatures exist only in the acoustic state snapshot are new paths; and paths whose path signatures exist only in the active path set are dying paths. For persistent paths, the path parameters of the persistent path are smoothed to the corresponding path parameters in the acoustic state snapshot in each audio block. For new paths, the frequency band energy gain is increased to the corresponding frequency band energy gain in the acoustic state snapshot within a preset fade-in duration. For dying paths, the frequency band energy gain is attenuated to 0 within a preset fade-out duration, and then the dying path is removed from the active path set.

[0008] Step 3: Within each audio block, the sound source signal is rendered in a binaural manner based on the smoothed path parameters of each path in the active path set to obtain the output binaural signal.

[0009] Furthermore, the hierarchical bounding box acceleration structure uses triangular facets as leaf nodes and consists of two levels: a top-level acceleration structure and a bottom-level acceleration structure, residing in the graphics processor's memory. The bottom-level acceleration structure of static geometry is built once during level loading and continuously reused in subsequent graphics frames. The bottom-level acceleration structure of dynamic geometry is refreshed in situ within the graphics frame where the pose or shape changes. The acoustic ray tracing task is submitted via an asynchronous computation queue, belongs to two independent command queues with the graphics ray tracing task, and accesses the hierarchical bounding box acceleration structure concurrently in read-only mode. It is scheduled to execute in the idle time slot after the hierarchical bounding box acceleration structure is updated in the current graphics frame and before the graphics ray tracing task starts.

[0010] Furthermore, the acoustic parameters for a predetermined number of frequency bands are obtained by querying a pre-established acoustic material mapping table. The acoustic parameters include the absorption coefficient and the transmission coefficient. The predetermined number of frequency bands is 8 octave bands. The sound propagation solution performs direct path solution and reflection path solution for each sound source. The direct path solution emits a direct detection ray from the sound source position to the listener position and collects the hit triangles by arbitrarily hitting to obtain the frequency band energy gain of the direct path. The reflection path solution emits a detection ray from the sound source position along the spherical Fibonacci grid direction. When the detection ray is reflected by the mirror and intersects with the detection sphere formed by the listener position as the center and the radius of the detection sphere, a crossing record is generated. The radius of the detection sphere is the larger of the preset lower limit of the radius and the adaptive radius. The adaptive radius is equal to twice the cumulative path length divided by the square root of the preset total number of detection rays.

[0011] Furthermore, for all traversal records with the same path signature, the frequency band energy gain is summed band by band, and the propagation delay and arrival direction are weighted and averaged according to the sum of the frequency band energy gains, and then merged into one effective sound propagation path. The expected value of the frequency band energy gain after merging is consistent with the energy attenuation of the corresponding mirror source under the mirror source model. The sound propagation solution also performs linear regression on the natural logarithm of the cumulative medium attenuation of all traversal records with the propagation delay to obtain the attenuation slope of each frequency band. Based on this, the reverberation time of each frequency band is obtained by extrapolating the predetermined energy attenuation in decibels, and is output along with the acoustic state snapshot.

[0012] Furthermore, matching is implemented using a hash table with path signatures as keys. The path parameters of the continuous path are exponentially smoothed and approximated block by block to the corresponding path parameters in the acoustic state snapshot. The frequency band energy gain of the new and dying paths is faded in and faded out respectively using the raised cosine envelope. When the number of paths in the active path set reaches the preset path number limit, the path with the smallest sum of frequency band energy gain is treated as a dying path. When the proportion of the number of dying paths to the total number of paths in the active path set exceeds the preset proportion threshold in each matching, the acoustic solution rate is temporarily increased and the fade-in and fade-out duration is shortened. The acoustic state snapshot carries a timestamp and is passed to the audio rendering thread through a double buffer in shared memory. During binaural rendering, the square root of the smoothed frequency band energy gain is used as the frequency band amplitude gain to filter the sound source signal in frequency band, sample according to the smoothed propagation delay, and then convolve the signal according to the head correlation transfer function selected according to the direction of arrival to obtain the output binaural signal.

[0013] This invention also provides an immersive directional sound effect simulation optimization system based on a virtual engine, including a graphics processor, a central processing unit, shared memory, an audio output device, and a memory; the graphics processor is equipped with a hardware ray tracing unit and an asynchronous computing queue; the memory stores a computer program; when the computer program is executed by the graphics processor and the central processing unit, it implements any of the above-mentioned immersive directional sound effect simulation optimization methods based on a virtual engine, wherein the sound propagation solution is executed on the graphics processor, the audio rendering thread runs on the central processing unit, the acoustic state snapshot is transferred from the graphics processor to the audio rendering thread via the shared memory, and the output binaural signal is played through the audio output device.

[0014] The immersive directional sound effect simulation optimization method and system based on a virtual engine of the present invention has the following beneficial effects:

[0015] Firstly, the acoustic ray tracing task and the graphics ray tracing task access the same bounding box acceleration structure. The acoustic solution directly borrows the acceleration structure built or incrementally updated for each graphics frame by the graphics rendering pipeline. It does not need to build and maintain a separate geometric representation for sound, nor does it need to repeatedly bear the construction overhead of the acceleration structure. Therefore, in dynamic scenes, it can update the sound field in real time with geometric changes, and significantly reduce the memory and computing power occupied by the acoustic solution. Furthermore, since the acoustic ray tracing task only accesses the acceleration structure in read-only mode, it can be executed concurrently with the graphics ray tracing task and is scheduled to run in the idle time slot after the acceleration structure is updated and before the graphics ray tracing task starts, further reducing the interference with graphics rendering. Secondly, the acoustic solution rate and the graphics frame rate are independent of each other. The acoustic state snapshots obtained from the acoustic solution establish a cross-snapshot correspondence through path signatures and active paths maintained by the audio rendering thread. Continuous paths are smoothly transitioned at the audio block level, and newly appearing and disappearing paths are gently faded in and out. This prevents the gap between the lower acoustic solution rate and the higher audio block rate from causing pops, directional jitter, and abrupt changes in timbre, giving the listener a continuous and stable sense of direction and distance. As the listener moves relative to the sound source, the smooth transition of propagation delay also naturally produces a Doppler shift, enhancing the realism of moving sound sources. Thirdly, when the scene undergoes drastic geometric changes, the method adaptively increases the acoustic solution rate and shortens the transition time accordingly. It quickly follows during periods of rapid change and saves computational power during periods of stability, balancing response speed and operational efficiency. Attached Figure Description

[0016] Figure 1 A schematic diagram illustrating the principle of hardware optical tracing propagation of a shared hierarchical bounding box acceleration structure provided in an embodiment of the present invention;

[0017] Figure 2 This is a timing diagram illustrating the staggered scheduling of acoustic solution tasks within graphics frames and the relationship between acoustic solution rate, graphics frame rate, and audio block rate, as provided in an embodiment of the present invention.

[0018] Figure 3 This is a schematic diagram illustrating the principle of reflection path solving provided in an embodiment of the present invention, wherein, Figure 3 (a) in the diagram is a schematic diagram illustrating the principle of receiving sphere ray tracing and passing record generation. Figure 3 (b) in the diagram illustrates the adaptive relationship between the detection sphere radius and the cumulative path length.

[0019] Figure 4 This is a schematic diagram illustrating the verification of energy normalization consistency provided in an embodiment of the present invention, wherein, Figure 4 (a) in the figure is a schematic diagram showing the relationship between the number of traversal records of a single mirror source and distance. Figure 4 (b) in the figure is a schematic diagram comparing the single-band energy gain after merging with the energy of the mirror source model;

[0020] Figure 5 This is a schematic diagram illustrating the principle of state smoothing provided in an embodiment of the present invention, wherein, Figure 5 (a) in the diagram illustrates the principle of path signature matching across snapshots and the determination of persistent paths, new paths, and extinct paths. Figure 5 (b) in the figure is a schematic diagram of the frequency band energy gain of the continuous path, the new path, and the extinction path as a function of time.

[0021] Figure 6 This is a schematic diagram of reverberation time estimation provided in an embodiment of the present invention, wherein, Figure 6 (a) in the diagram illustrates the relationship between the logarithmic value of the cumulative medium attenuation and the linear regression of propagation delay. Figure 6 (b) in the diagram is a schematic diagram of the reverberation time distribution of each octave band. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The specific embodiments described herein are used to explain the implementation of this invention; processes not detailed in the embodiments can be performed in accordance with conventional methods in the art.

[0023] The implementation process of this invention will be described in detail below using a specific scenario running in a virtual engine. This scenario is an indoor hall with an openable metal door. The hall contains several walls, floors, and glass curtain walls of varying materials. A continuously playing speaker serves as the sound source. The user moves around the hall from a first-person perspective, turning their head. The metal door opens or closes as the user passes through, continuously changing the geometric connectivity of the hall. This scenario, combining static walls and dynamic doors with significant reflections and occlusions, is chosen to fully demonstrate the invention's ability to solve sound fields and render artifact-free images under dynamic geometry.

[0024] At the hardware level, the above scenario runs on a device equipped with a graphics processor that supports hardware ray tracing. The graphics rendering pipeline and acoustic solving share the hardware ray tracing unit on the graphics processor. Audio rendering is performed on a separate thread of the central processing unit, and the graphics processor and this separate thread exchange data through a shared memory block. This invention is not limited to a specific model of graphics processor; any graphics processor that provides a hardware ray tracing unit and an asynchronous computation queue, and allows concurrent read-only intersection operations to be initiated on the same acceleration architecture, can be used to implement this invention.

[0025] During the scene loading phase, the virtual engine's main thread completes the registration of sound sources and listeners. The sound source registration information includes the sound source's position in the world coordinate system and the sound source signal, which is the mono waveform output by the speaker sample by sample. The listener registration information includes the listener's position and orientation, with the orientation represented by a quaternion. Quaternions are used instead of Euler angles to record orientation because they do not suffer from the gimbal lock problem inherent in Euler angles when describing continuous rotations, and the interpolation between the two orientations is smooth and unique, providing a stable rotational basis for subsequently transforming the sound's arrival direction to the listener's local coordinate system.

[0026] Meanwhile, the virtual engine establishes an acoustic material mapping table for each type of graphic material appearing in the scene. The acoustic material mapping table is indexed by the graphic material identifier, which is an integer number already assigned to each triangle facet by the virtual engine during rendering. This invention directly borrows this identifier instead of creating a separate acoustic identifier, thus avoiding the need to maintain an independent numbering system and its synchronization burden on the acoustic side. For each graphic material identifier, the acoustic material mapping table records its acoustic parameters in a predetermined number of frequency bands. These acoustic parameters include the absorption coefficient and transmission coefficient: the absorption coefficient describes the proportion of energy absorbed when sound waves are reflected from the material surface, and the transmission coefficient describes the proportion of energy retained when sound waves pass through the material. In this embodiment, the predetermined number of frequency bands is taken as eight octave bands. The center frequencies of the eight octave bands increase by octaves starting from 62.5 Hz, namely 62.5 Hz, 125 Hz, 250 Hz, 500 Hz, 1000 Hz, 2000 Hz, 4000 Hz, and 8000 Hz, covering the main frequency range perceptible to the human ear. The reason for dividing the frequency bands by octaves rather than equal intervals is that the human ear's frequency discrimination approximately follows a logarithmic law. Octave division ensures that each frequency band has similar auditory importance, allowing for more balanced coverage of the entire frequency range with fewer bands. For example, a brick wall in a hall has a sound absorption coefficient of approximately 0.03 in the low-frequency range and approximately 0.07 in the high-frequency range, exhibiting a slightly stronger high-frequency absorption characteristic. The acoustic material mapping table automatically generates initial values ​​based on the category labels of the graphic materials during the asset import stage and allows for manual revision in the editor.

[0027] After scene registration, the frame-by-frame execution phase begins. The graphics rendering pipeline constructs or incrementally updates the hierarchical bounding box acceleration structure for scene geometry in each graphics frame. The hierarchical bounding box acceleration structure, with triangular facets as leaf nodes, consists of a top-level acceleration structure and a bottom-level acceleration structure, residing in the graphics processor's memory. The bottom-level acceleration structure describes the spatial hierarchy of triangular facets within a single geometry, while the top-level acceleration structure describes the position and orientation of each geometry instance in the scene. For static walls, floors, and curtain walls in the lobby, their bottom-level acceleration structure is constructed once during level loading and reused in subsequent graphics frames without reconstruction, because the triangular facets of static geometry do not move relative to themselves. For dynamic geometry such as openable metal doors, their bottom-level acceleration structure is refreshed in situ within the graphics frame where the door's pose or shape changes; that is, the bounding box boundaries are updated locally within the existing memory space, avoiding the overhead of reallocating memory and rebuilding from scratch. This two-level division and static / dynamic separation concentrates the maintenance overhead of the acceleration structure on the small number of truly changing dynamic geometries.

[0028] In this invention, the acoustic solution does not construct a new geometric representation, but directly reuses the hierarchical bounding box acceleration structure constructed or updated for graphics rendering. For example... Figure 1 As shown, the graphics rendering pipeline builds or updates a hierarchical bounding box acceleration structure in the graphics processor's memory. The graphics ray tracing task reads this acceleration structure to complete the rendering. Similarly, the acoustic ray tracing task also accesses this acceleration structure. Acoustic solving is performed periodically at an acoustic solver rate, which is lower than the graphics frame rate, and the acoustic solver rate and graphics frame rate are set independently. The reason for decoupling the two is that screen refresh requires a sufficiently high frame rate to ensure visual smoothness, while the sound field differs little between adjacent moments when the scene does not change drastically. Solving the sound field at a lower frequency is sufficient to meet auditory needs, thus allocating more of the graphics processor's computing power to screen rendering. In this embodiment, the graphics frame rate is approximately 90 frames per second, and the acoustic solver rate is approximately 15 times per second, meaning that acoustic solving is performed approximately once every 6 graphics frames.

[0029] To minimize interference with image rendering, acoustic ray tracing tasks are submitted to the graphics processor's hardware ray tracing unit via an asynchronous computation queue, belonging to two independent command queues separate from the graphics ray tracing tasks. Since acoustic solving does not modify the accelerated structure's contents while traversing it, both acoustic and graphics ray tracing tasks access the hierarchical bounding box accelerated structure in read-only mode, allowing them to execute concurrently without data corruption. Furthermore, acoustic ray tracing tasks are scheduled to execute in the graphics rendering pipeline's idle time slot after the hierarchical bounding box accelerated structure update for the current graphics frame and before the graphics ray tracing task for the current frame starts. This time slot is chosen based on two points: first, the accelerated structure must be accessed after its update to ensure that the acoustic solution obtains the latest geometry of the current frame; second, there is often an idle time slot between the graphics processor completing the accelerated structure update and starting graphics ray tracing, during which computation units are not fully occupied. Scheduling acoustic solving during this time effectively utilizes the idle period of image rendering to complete the sound field solution, minimizing the impact on the frame rate.

[0030] Figure 2 The temporal relationship between the three rates and the scheduling position of the acoustic solution within the graphics frame are shown. Figure 2 The top row is the timeline of the graphics frames, with each frame containing accelerated structure updates, idle time slots, and graphics ray tracing in sequence. The middle row is the timeline of the acoustic solution cycle. Acoustic ray tracing executes within the idle time slots of the graphics frames, and its cycle is longer than that of the graphics frames, reflecting that the acoustic solution rate is lower than the graphics frame rate. The bottom row is the timeline of the audio blocks, which are densely packed and have a higher rate than the acoustic solution rate. Figure 2 As can be seen, the acoustic solution occurs only once every few graphic frames, while the audio block is continuously output multiple times between two acoustic solutions. It is this rate difference that requires smoothing of the sound field at the audio block level, which will be discussed later. Figure 5 Detailed explanation.

[0031] Within each acoustic solution cycle, the sound propagation solution performs direct path and reflection path calculations for each sound source, followed by path signature generation and signature merging, and reverberation time estimation. Let's first explain the direct path calculation. The direct path calculation emits a direct detection ray from the sound source location to the listener location. This direct detection ray uses an arbitrary hit traversal mode to collect all hit triangles along its path, rather than stopping after encountering the first hit triangle. The reason for using arbitrary hit traversal is that the direct sound may pass through several semi-transparent barriers, such as glass curtain walls or half-open doors, before reaching the listener. Only by collecting all hit triangles along the path can the energy attenuation caused by each penetration be correctly accounted for. For each hit triangular facet collected, the frequency band transmission coefficient is obtained by querying the acoustic material mapping table based on the graphic material identifier it carries. Then, the frequency band transmission coefficients of all hit triangular facets are multiplied together band by band. When the direct detection ray does not intersect with any triangular facet, that is, the number of hit triangular facets is 0, the multiplication result is taken as 1, indicating that there is no obstruction between the sound source and the listener, and the direct sound is not attenuated due to penetration.

[0032] After obtaining the penetration multiplication result, the direct path constitutes an effective sound propagation path, whose frequency band energy gain is determined by the following formula, where the energy gain of the j-th frequency band is equal to the product of the penetration multiplication result of that frequency band, the air absorption factor of that frequency band, and the free field propagation attenuation:

[0033]

[0034] In the above formula, the symbols have the following meanings. The left side of the equal sign represents the direct path energy gain of the j-th frequency band, where j is the frequency band number, ranging from 1 to a predetermined number of frequency bands; the multiplication sign gives the product of the transmission coefficients of all hit triangular facets in that frequency band; the exponent term is the air absorption factor of that frequency band, which takes the natural constant as the base and the product of the negative air absorption factor and the total propagation path length as the exponent. The air absorption factor is pre-configured based on ambient temperature, relative humidity, and atmospheric pressure, and the total propagation path length is taken as the straight-line distance from the sound source to the listener for the direct path; the fractional term is the free-field propagation attenuation, with the denominator being the product of pi, the value 4, and the square of the total propagation path length, and the numerator being 1. The reason for introducing the air absorption factor is that high-frequency components of sound waves are significantly absorbed when propagating over long distances in the air. The farther the distance and the higher the frequency, the more obvious the absorption. If omitted, the high frequencies of distant sound sources will be distorted. Free-field propagation attenuation describes the decrease in energy of a sound wave as it propagates outward as a spherical wave. When the same amount of sound power is distributed across a sphere with an area proportional to the square of the distance, the energy per unit area is inversely proportional to the square of the distance. Therefore, the denominator includes the square of the total propagation path length. Multiplying penetration, air absorption, and geometric diffusion, since these three attenuation mechanisms act independently on the same sound wave, the combined effect of these independent attenuations is the product of their respective attenuation factors.

[0035] In parallel with the direct path solution, the sound propagation solution performs reflection path solution for each sound source. The reflection path solution emits a number of probe rays equal to the preset total number of probe rays along the spherical Fibonacci grid direction from the sound source location. The reason for emitting along the spherical Fibonacci grid direction, rather than random directions or latitude and longitude grid directions, is that the Fibonacci grid can generate a nearly uniform point distribution on the sphere using a simple analytical formula. This avoids the uneven density that occurs in random directions and the excessive density at the poles that occurs in latitude and longitude grids. As a result, the finite number of probe rays can cover each emission direction as evenly as possible, reducing the energy deviation caused by uneven directional sampling.

[0036] The principle of reflection path solving is as follows: Figure 3 As shown in (a) of the diagram. The probe ray emitted from the sound source propagates through the scene. After each probe ray hits a triangular facet, it continues to propagate along the mirror reflection direction. That is, with the surface normal at the hit point as the axis, the incident direction is mirrored as the reflection direction, just like a ray encountering a mirror. It undergoes at most a preset maximum number of reflections; after exceeding this limit, it stops tracking to restrict the computational depth of a single ray. At each hit point, the hit shader reads the graphic material identifier of the hit triangular facet and completes an acoustic material mapping table lookup to obtain the sound absorption coefficient of that reflection in each frequency band. Then, it successively multiplies the reflection retention amount obtained by subtracting the sound absorption coefficient from 1. This successively multiplied intermediate amount of frequency band reflection attenuation is stored in the ray load data and is passed between adjacent reflections with the probe ray. This ensures that for each subsequent reflection, only the reflection retention amount of the current facet needs to be multiplied into the existing intermediate amount, avoiding backtracking the entire reflection chain for recalculation when generating the path.

[0037] To determine whether a probe ray has reached the listener, this invention constructs a detection sphere with the listener's position as the center and the detection sphere's radius as the sphere. During the propagation of the probe ray, it detects whether the propagation segment intersects the detection sphere. When the propagation segment of the probe ray intersects the detection sphere after at least one reflection, a crossing record is generated. After recording, the probe ray continues to propagate, so that the same probe ray may cross the detection sphere again and be recorded after subsequent higher-order reflections. A crossing is only counted after at least one reflection to separate reflected sound from direct sound: direct sound is already processed separately by the direct path solution; if probe rays that have not yet reflected are allowed to cross the detection sphere and be recorded, they will be counted twice with the direct path sound in the same amount of sound energy. Figure 3 As shown in (a), after one reflection from the wall, part of the detection rays emitted from the sound source pass through the detection sphere centered on the listener, leaving a crossing point and generating a crossing record; the other part of the detection rays do not intersect the detection sphere due to directional deviation, and therefore do not generate a crossing record.

[0038] Each crossing record includes propagation delay, direction of arrival, and frequency band energy gain. The crossing point is the point on the propagation line segment closest to the listener's position, i.e., the position closest to the center of the sphere within the segment of the propagation line segment. This point represents the time and direction of the sound wave passing through the listener's neighborhood, best reflecting the actual state of the sound wave passing through the listener. The propagation delay is equal to the cumulative path length at the crossing point divided by the speed of sound. The cumulative path length is the path length that the probe ray has traveled from the sound source, i.e., the total distance the sound wave travels along the broken line from the sound source to the crossing point; dividing by the speed of sound gives the time required for the reflected sound to reach the listener. The direction of arrival is the unit vector opposite to the direction of the probe ray's propagation at the crossing moment, stored in the world coordinate system. The reason for using the opposite direction is that, from the listener's perspective, the sound comes from the opposite direction of the probe ray's origin. Storing it in the world coordinate system makes the direction of arrival independent of the listener's current orientation. During binaural rendering, it is transformed to the listener's local coordinate system based on the listener's latest orientation, thus eliminating the need to re-solve the sound field when the listener turns their head.

[0039] The energy gain of each traversal record in the j-th frequency band is determined by the following formula, which is equal to the air absorption factor of that frequency band divided by the normalized area, and then multiplied by the product of the reflection attenuation of that frequency band:

[0040]

[0041] The meanings of the quantities in the above formula are as follows: The left side of the equation represents the energy gain of the crossing record in the j-th frequency band; the exponent term of the numerator is the air absorption factor of that frequency band, which is defined the same as in the direct path, except that the total propagation path length here is taken as the cumulative path length at the crossing point; the denominator is the normalized area, which is equal to the product of the preset total number of detection rays, pi, and the square of the radius of the detection sphere. The preset total number of detection rays is recorded as the number of rays; the multiplication sign gives the reflection attenuation product, that is, the product of the reflection retention coefficients corresponding to each reflection before crossing in that frequency band. The reflection retention coefficient is equal to the difference between the value 1 and the absorption coefficient of the triangular facet hit by the corresponding reflection in that frequency band. The normalized area appearing here is the key to the correct energy accounting of this invention, and its cause is explained below.

[0042] The method of statistically analyzing reflected sound energy by emitting probe rays from a sound source and receiving them with a detection sphere essentially estimates a continuous sound energy flow using a finite number of discrete rays. Imagine the sound source radiating uniformly throughout space, its power evenly distributed across the predetermined number of probe rays. The number of rays per unit solid angle is proportional to the predetermined number of probe rays and inversely proportional to the square of the distance. The cross-section of the detection sphere in the direction of the sound wave is a disk with a radius equal to the radius of the detection sphere, and its area is the product of pi and the square of the detection sphere's radius. Therefore, the number of rays passing through the detection sphere is equal to the product of the ray flux per unit area and the area of ​​this cross-section. The sound energy represented by a single ray should be normalized to its corresponding area fraction, which is the reciprocal of the product of the predetermined number of probe rays, pi, and the square of the detection sphere's radius—the normalized area in the denominator of the above equation. Dividing by the normalized area precisely restores the count of the finite number of discrete rays to the intensity of the continuous sound energy flow, making the estimation unbiased.

[0043] To verify the correctness of the above normalization, consider a mirror sound source determined by a single reflection. In geometrical acoustics, a single mirror reflection is equivalent to placing a mirror sound source on the other side of the mirror, and the reflected sound can be considered to travel in a straight line from this mirror sound source. For a fixed detection sphere radius, the expected number of probe rays passing through the detection sphere is proportional to the product of the preset total number of probe rays and the square of the detection sphere radius, and inversely proportional to the square of the distance from the mirror sound source to the listener. The energy of these passing records is accumulated one by one according to the above formula, and the expected sum is exactly equal to the energy that the mirror sound source should receive due to free-field propagation attenuation and reflection attenuation. This self-consistent relationship can be written as follows: when the detection sphere radius is equal to the adaptive radius, the expected value of the number of passing records corresponding to a single mirror sound source is a constant value as shown in the following formula:

[0044]

[0045] Figure 4 (a) in the figure provides a numerical verification of this relationship. Figure 4 In (a), the horizontal axis represents the distance from the sound source to the listener, and the vertical axis represents the number of traversal records corresponding to a single mirror sound source. The dashed line and hollow circle represent the theoretical expectation and ray statistics when the detection sphere radius is fixed. It can be seen that as the distance increases, the number of traversal records decreases rapidly inversely proportional to the square of the distance. At a distance, only a few or even no rays pass through the detection sphere, and the reflected sound is therefore severely undersampled. The solid line and solid triangle represent the theoretical expectation and ray statistics when the detection sphere radius is adaptive. The expected number of traversal records does not change with the distance, so stable sampling can be obtained at both near and far distances. Figure 4(b) further compares the combined single-band energy gain with the energy of the mirror source model on the logarithmic vertical axis and the total propagation path length on the horizontal axis. It can be seen that the energy obtained by ray statistical merging coincides with the energy given by the mirror source model at various distances, indicating that the normalized energy accounting of the present invention is consistent with the mirror source model in the expected sense, and the energy estimation is correct.

[0046] The aforementioned problem of undersampling at long distances due to a fixed radius prompted this invention to make the detection sphere radius adaptive with distance. For example... Figure 3 As shown in (b), the detection sphere radius is the larger of the preset lower radius limit and the adaptive radius. The adaptive radius is equal to twice the cumulative path length divided by the square root of the preset total number of detection rays. When the cumulative path length is small, the adaptive radius is less than the preset lower radius limit, so the preset lower radius limit is used to avoid the detection sphere being too small and missing detections at close range. When the cumulative path length increases to a certain extent, the adaptive radius exceeds the preset lower radius limit and increases linearly with distance, so the adaptive radius is used. Figure 3 In (b), the horizontal axis represents the cumulative path length, the vertical axis represents the detection sphere radius, the dashed line represents the adaptive radius, the dotted line represents the lower limit of the preset radius, and the solid line represents the detection sphere radius obtained by taking the larger of the two. The intersection of the two lines is the point where the adaptive radius and the lower limit of the preset radius are equal. The detection sphere radius to the right of this point increases linearly with distance. The adaptive radius is made proportional to the distance and the above coefficient is taken because: when selected in this way, regardless of the distance of the mirror sound source, the expected number of rays passing through the detection sphere is always 1. Figure 4 As shown by the solid line in (a), this avoids missing detections at distant locations while preventing the mixing of closely spaced different reflection paths into the same detection sphere due to an excessively large radius, thus balancing detection rate and directional resolution. Furthermore, since detection and normalization use the same detection sphere radius, adaptive changes in the detection sphere radius do not alter the energy accounting in the above formula, and the expected energy after merging remains consistent with the mirror source model.

[0047] After generating the traversal record, this invention generates a path signature for each direct path and each traversal record to identify the geometric route traversed by the sound wave. The path signature is constructed by concatenating the sound source identifier, the geometric instance identifiers of the hit triangular facets, and the facet indices in the order of reflection. The path signature for a direct path consists of the sound source identifier and a preset reserved value. Placing the sound source identifier at the beginning of the path signature distinguishes paths from different sound sources, preventing sounds from different sources traveling along the same geometric route from being misjudged as the same path. Concatenating the geometric instance identifiers and facet indices in the order of reflection uniquely records which triangular facets of which geometry the sound wave passed through. Two sound waves traveling through the exact same reflection sequence will have identical path signatures; any difference in the reflection facets will result in different path signatures. For direct paths, since they do not experience reflection, a preset reserved value occupies the reflection sequence position, distinguishing its path signature from any reflected path.

[0048] Since multiple detection rays emitted along the Fibonacci lattice points on the sphere often pass through the detection sphere sequentially after passing through the same reflection sequence, they correspond to the same geometrically significant reflection path and thus have the same path signature. This invention merges all traversal records with the same path signature: their frequency-band energy gains are summed band by band; propagation delay and arrival direction are averaged by weighting the sum of the frequency-band energy gains; the arrival direction is weighted and averaged, then normalized to a unit vector, and merged into one effective sound propagation path. The energy is summed band by band because these traversal records represent the same reflection path being repeatedly sampled by multiple rays, and their energy should be superimposed to restore the total energy of the path. The aforementioned normalization area ensures that the expected value after superposition is consistent with the mirror sound source model. The delay and direction are weighted and averaged by energy because samples with higher energy are closer to the center of the reflection path and should therefore have a greater weight, resulting in a more representative delay and direction that closely resembles the actual reflection path. The arrival direction is weighted and averaged, then normalized because the weighted sum of several unit vectors is generally no longer a unit length and must be normalized to restore its meaning as a direction. Each effective sound propagation path obtained by merging has path parameters including propagation delay, direction of arrival, and frequency band energy gain. The expected value of the frequency band energy gain after merging is consistent with the energy attenuation of the corresponding mirror source under the mirror source model.

[0049] In addition to direct and reflection paths, the sound propagation solution also performs reverberation time estimation to characterize the overall attenuation rate of dense late reflections in the hall. This invention performs a linear regression of the natural logarithm of the cumulative media attenuation in each frequency band of all traversing records with propagation delay to obtain the attenuation slope for each band. The cumulative media attenuation in each frequency band is equal to the product of the reflection attenuation of the corresponding band and the air absorption factor of the corresponding band; that is, the amount remaining after removing the normalized area term from the traversing record energy. Taking only the product of the reflection attenuation and the air absorption factor, instead of directly taking the traversing record energy, is based on the following considerations: the traversing record energy also contains a free-field propagation attenuation term, which decreases inversely with the square of the distance, reflecting the geometric effect of sound wave diffusion outwards, and is independent of the hall's own sound absorption strength; if this term were included in the regression, the regression slope would be contaminated by geometric diffusion, and the resulting reverberation time would drift with the distance between the sound source and the listener, losing a stable characterization of the room's reverberation. After eliminating the attenuation from free-field propagation, the remaining product of reflection attenuation and air absorption factor only reflects how much sound wave is absorbed during reflection and passage through the medium. Its logarithm decreases approximately linearly with the propagation delay, and the slope is determined only by the sound absorption of the material and air absorption, which is a true reflection of the rate of reverberation decay.

[0050] like Figure 6As shown in (a), after taking the natural logarithm of the cumulative medium attenuation in each frequency band and plotting it against the propagation delay, the scatter points for the same frequency band are approximately distributed along a straight line. A linear regression of this distribution yields the attenuation slope for that frequency band. A steeper slope indicates faster energy decay and shorter reverberation, while a gentler slope indicates longer reverberation. Extrapolating the time required for a predetermined energy attenuation in decibels from the attenuation slope of each frequency band yields the reverberation time for that band. The predetermined decibel is 60, meaning the reverberation time is the time it takes for the energy to decay from its initial value to one millionth of its original value. This relationship can be expressed as the following formula, where the reverberation time equals 60 divided by the attenuation in decibels per second calculated based on the attenuation slope:

[0051]

[0052] In the above formula, the left side of the equation represents the reverberation time of the j-th frequency band, in seconds; the numerator is a predetermined decibel level, taken as 60; the denominator is the decibel attenuation per second of the j-th frequency band calculated according to its attenuation slope, which is derived from... Figure 6 The attenuation slope obtained in (a) is multiplied by the speed of sound and converted to an attenuation rate in decibels. The steeper the attenuation slope, the greater the attenuation in decibels per second and the shorter the reverberation time. Figure 6 (a) shows the scatter plots of the low-frequency band and the high-frequency band and their regression lines with hollow circles and solid triangles, respectively. It can be seen that the regression line of the high-frequency band is steeper and corresponds to a shorter reverberation time, which is consistent with the physical fact that high frequencies are absorbed more strongly in air and materials. Figure 6 (b) shows the reverberation time of each of the eight octave bands in a bar chart. The horizontal axis represents the center frequency of the octave band, and the vertical axis represents the reverberation time. The reverberation time decreases from low frequency to high frequency, fully depicting the reverberation characteristics of the hall in each frequency band. The regression is performed on the acoustic solution side, taking all the traversal records at once, rather than estimating each path on the audio side, because the number of traversal records is far greater than the number of merged paths, and the more samples, the more robust the regression. The resulting reverberation time of each frequency band is output as a small scalar along with the acoustic state snapshot for direct use on the audio side.

[0053] At this point, the solution results for one acoustic solution cycle are complete. This invention packages the sound source, path signature, and path parameters of all valid sound propagation paths within the current acoustic solution cycle, along with the reverberation time of each frequency band and the timestamp of the generation time, into a single acoustic state snapshot. Since the acoustic solution is performed in the graphics processor's video memory, the acoustic state snapshot is copied from the graphics processor's video memory to shared memory via a readback buffer, and then passed to the audio rendering thread through a dual buffer in shared memory. The dual buffer has two storage blocks: the graphics processor writes new acoustic state snapshots to one block, while the audio rendering thread reads completed acoustic state snapshots from the other block; after the acoustic state snapshot is written, the dual buffer switches its read / write pointers as a whole, allowing the read / write sides to access different storage without interference. This design avoids the audio rendering thread reading half-written acoustic state snapshots: Since the graphics processor and audio rendering thread run asynchronously, if they share a single memory block, reads and writes might occur simultaneously on the same block, leading to inconsistent data where latency has been updated but energy has not. Double buffering and overall switching ensure that each read is a complete and consistent acoustic state snapshot. Furthermore, the acoustic state snapshot carries a timestamp of its creation time, and the audio rendering thread only adopts the acoustic state snapshot with the latest timestamp to prevent the misuse of older solution results due to scheduling fluctuations.

[0054] The audio rendering thread runs continuously in units of audio blocks. The audio block rate is equal to the audio sampling rate divided by the number of audio block samples, and is higher than the acoustic solution rate. In this embodiment, the audio sampling rate is 48,000 samples per second, and the number of audio block samples is 256, so the audio block rate is approximately 187 blocks per second, which is much higher than the acoustic solution rate of 15 times per second. The audio rendering thread maintains a set of active paths. Each path in the active path set records its associated sound source, as well as the smoothed propagation delay, direction of arrival, and frequency band energy gain. Since the acoustic solution rate is much lower than the audio block rate, if the path parameters are directly replaced with new values ​​each time a new acoustic state snapshot is received, the jump in path parameters between adjacent solutions will sound like pops, sudden changes in direction, and abrupt changes in timbre. Therefore, the path parameters must be smoothly transitioned at the audio block level. This is the core of this invention to decouple the solution rate and output rate without producing artifacts.

[0055] Whenever a new acoustic state snapshot is adopted, the audio rendering thread matches the path signature set of the acoustic state snapshot with the path signature set of the active path set. This matching is achieved using a hash table with path signatures as keys, thus quickly finding the correspondence between the two sets at a time cost proportional to the number of paths. For example... Figure 5As shown in (a), the matching classifies paths into three categories: persistent paths are those whose path signatures exist in both sets; newly emerging paths are those whose path signatures exist only in the acoustic state snapshot; and perishing paths are those whose path signatures exist only in the active path set. A persistent path represents the same geometric reflection path that exists in both solutions, although its time delay, direction, and energy may change. A newly emerging path represents a reflection path that appears due to scene changes or sound source movement. A perishing path represents a reflection path that existed previously but has now disappeared due to occlusion or geometric changes. Using path signatures as identifiers to establish this correspondence allows smoothing to be performed between paths with the same geometric meaning, without incorrectly smoothing two unrelated reflection paths together.

[0056] For a sustained path, the propagation delay, direction of arrival, and frequency-division energy gain in the acoustic state snapshot are set to the corresponding target values. Then, for each subsequent audio block, the propagation delay, direction of arrival, and frequency-division energy gain held by the sustained path are smoothed block by block towards the corresponding target values ​​using the following formula: This smoothing is exponential smoothing.

[0057]

[0058] The meanings of the quantities in the above formula are as follows: The left side of the equation is the smoothed value of the nth audio block, where n is the audio block number; the first term on the right side is the smoothed value of the (n-1)th audio block; the value in parentheses is the difference between the target value and the smoothed value of the (n-1)th audio block; this difference is multiplied by the smoothing coefficient and added to the smoothed value of the previous audio block to obtain the smoothed value of the current audio block. The smoothing coefficient is given by the second formula, which is equal to the difference between the value 1 and an exponential term. This exponential term has a base of the natural constant and an exponent of the ratio of the negative audio block duration to the preset smoothing time constant; the audio block duration is the time length corresponding to one audio block, and the preset smoothing time constant is the time constant that controls the smoothing speed, which is set for three types of path parameters: propagation delay, direction of arrival, and frequency band energy gain. This exponential smoothing is employed because it is equivalent to first-order low-pass tracking of the target value. Each audio block advances towards the target value by a fixed percentage of the current gap, thus gradually approaching the target value exponentially without overshooting or oscillating. A larger preset smoothing time constant results in a smoother approach, while a smaller one results in greater sensitivity. By setting time delay, direction, and energy separately, a suitable trade-off between stability and sensitivity can be achieved. The three components of the direction of arrival are smoothed according to the above formula and then renormalized into unit vectors to retain their meaning as directions.

[0059] The smoothing of propagation delay is achieved through a fractional delay line. The fractional delay line is a delay buffer that can be read at non-integer sample positions. Within each audio block, the read position transitions linearly sample-by-sample from the read position of the previous audio block to the smoothed value of the current audio block. As the read position moves continuously, the time at which the sound is read within the buffer changes continuously, resulting in a Doppler shift: when the read position moves further back as the time delay increases, it is equivalent to the sound source moving further away, causing the pitch to decrease; conversely, the pitch increases. Thus, the Doppler effect is not a separate additive effect, but rather arises naturally with the smoothing of the time delay, eliminating the need for separate pitch shifting and ensuring strict consistency with the time delay changes.

[0060] For newly emerging paths, their crossband energy gain is added to the active path set starting from a value of 0, and then increased to the corresponding target value within a preset fade-in duration. The propagation delay and direction of arrival are directly taken from the path parameters in the acoustic state snapshot starting from the first audio block. For fading paths, their crossband energy gain decays to 0 within a preset fade-out duration, and then the fading path is removed from the active path set. The propagation delay and direction of arrival maintain their final smooth values. The energy of newly emerging paths fades in from 0 rather than appearing abruptly, and the energy of fading paths gradually fades out to 0 rather than disappearing suddenly, both to avoid abrupt changes in the waveform caused by the appearance and disappearance of paths, resulting in popping sounds. The delay and direction of newly emerging paths directly take the target values ​​without smoothing, because there are no historical values ​​to transition from; fading in from 0 energy itself ensures smoothness. The delay and direction of fading paths maintain their final smooth values, because their energy is approaching 0, and the values ​​no longer affect the listening experience. Both the boost and attenuation of crossover energy gain are performed using the raised cosine envelope. The raised cosine envelope changes monotonically within its corresponding duration, with its first and last derivatives being zero. The raised cosine envelope is chosen because its tangent is horizontal at both the start and end points, and the rate of energy change increases smoothly from 0 and then smoothly returns to 0. Compared to linear fade-out, which leaves no sharp angles at either end, this results in a smoother, more imperceptible sound. Figure 5 As shown in (b), the frequency band energy gain of the continuous path approaches the target value with an exponential smooth curve, the frequency band energy gain of the new path fades in from 0 to the target value with the raised cosine envelope, and the frequency band energy gain of the dying path fades out from the current value to 0 with the raised cosine envelope. The energy of the three types of paths changes continuously without jumps on the time scale of the audio block.

[0061] To constrain the computational overhead on the audio side, this invention sets an upper limit on the size of the active path set. When the number of paths in the active path set reaches the preset upper limit, the path with the smallest total frequency band energy gain in the active path set is treated as a dying path before being added to the new path set. The reason for eliminating the path with the smallest total energy is that this path contributes the least to the final listening experience, and the auditory loss introduced by smoothly fading it out is minimal. The freed-up slots can be used for new paths with stronger and more important energy to enter the active path set. In this embodiment, the preset upper limit for the number of paths is set to 64, meaning that a maximum of 64 reflection paths can participate in rendering simultaneously. This is sufficient to present rich early reflections while limiting the per-path processing overhead of each audio block to a controllable range.

[0062] In addition to the path number limit, this invention also adaptively adjusts the acoustic solution rate according to the drastic changes in the scene. The audio rendering thread counts the proportion of the number of eliminated paths to the total number of active paths in each match. When this proportion exceeds a preset threshold, the acoustic solution rate is temporarily increased to the preset upper limit, and the preset fade-in duration and preset fade-out duration are shortened simultaneously, until the proportion falls back to within the preset threshold, at which point the acoustic solution rate, preset fade-in duration, and preset fade-out duration are restored to their original values. The proportion of disappearing paths is used as the criterion for dramatic scene changes. When metal doors in the lobby open and close rapidly or users pass through quickly, a large number of reflected paths disappear simultaneously within a single solution interval, causing the proportion of disappearing paths to surge. Increasing the acoustic solver rate at this time allows the sound field to follow geometric changes more quickly, while shortening the fade-in and fade-out duration allows for a faster transition between old and new paths, avoiding noticeable sound and image lag during dramatic changes due to slow transitions. Once the scene stabilizes, the proportion of disappearing paths drops, and a lower acoustic solver rate and a longer transition duration are resumed to maintain the sound field with minimal overhead, thus automatically achieving a balance between response speed and operational overhead. In this embodiment, the preset proportion threshold is 0.3, and the preset upper limit of the acoustic solver rate is 60 times per second.

[0063] Within each audio block, the audio rendering thread performs binaural rendering on the sound source signal based on the smoothed path parameters of each path in the active path set, resulting in the output binaural signal. For each path in the active path set, the square root of its smoothed cross-band energy gain is used as the cross-band amplitude gain. This amplitude gain is then applied to the sound source signal of the path's corresponding bandpass filter bank through cross-band filtering and weighted summation. The square root of the energy gain is used as the amplitude gain because energy is proportional to the square of the amplitude, and the amplitude, not the energy, directly affects the waveform. Therefore, the square root must be taken to convert the energy gain into amplitude gain to ensure the correct energy ratio of each path after superposition. By using the corresponding sound source signal for each path, it is ensured that in the case of multiple sound sources, each path filters the waveform of its corresponding sound source without confusion.

[0064] After completing the frequency band amplitude adjustment, the filtering result is fed into the fractional delay line and sampled according to the smoothed propagation delay of the path, so that the sound of the path obtains the correct arrival time and the aforementioned Doppler frequency shift. Subsequently, the arrival direction of the smoothed path, stored in the world coordinate system, is transformed to the listener's local coordinate system through the transpose of the rotation matrix corresponding to the listener's orientation. The listener's orientation is obtained when registering the listener and represented as a quaternion. The advantage of recording the arrival direction in the world coordinate system and then transforming it to the listener's local coordinate system during rendering is that the arrival direction obtained from the sound field solution is fixed relative to the scene. When the listener only turns their head without moving, the arrival direction in the world coordinate system of each path does not change. Only the coordinate transformation needs to be performed again with the latest orientation of the listener. There is no need to resolve the sound field due to head turning, which is especially important for immersive applications with frequent head turning. The transpose of the rotation matrix is ​​taken because the vector is transformed from the world coordinate system to the local coordinate system that rotates with the listener's orientation using the inverse of the orientation rotation matrix, and the inverse of the rotation matrix is ​​equal to its transpose.

[0065] After obtaining the representation of the arrival direction in the listener's local coordinate system, a head-related transfer function (HRT) is selected from the HRT library based on the nearest neighbor of azimuth and pitch angles according to the transformed direction. The sampled results are then convolved to obtain the binaural signal components of the path. The HRT characterizes the time difference, intensity difference, and spectral coloration between the two ears due to scattering and obstruction by the head, auricle, and torso when sound travels from a certain direction in space to the left and right ears. It is the physical basis for the human ear to judge the direction of sound. Selecting the corresponding HRT for each path based on its arrival direction and convolving it assigns the correct spatial orientation perception to that path. The nearest neighbor selection method based on azimuth and pitch angles simply selects the closest direction to the target direction from the discrete measurement directions in the transfer function library, which is simple to implement and has a sufficiently small error when the library is dense enough. Finally, the binaural signal components of all paths in the active path set are superimposed, and then a parameterized late reverberation signal is superimposed to obtain the output binaural signal, which is then played through an audio output device.

[0066] The parameterized late reverberation signal is generated by a feedback delay network. This network is a set of delay lines with feedback, capable of synthesizing dense and naturally decaying late reverberation with relatively low computational cost. The reverberation time of each frequency band in the feedback delay network is smoothed towards the reverberation time of each frequency band in the acoustic state snapshot for each audio block. That is, the reverberation times of each frequency band obtained through regression and output with the acoustic state snapshot are used as the target reverberation time of the feedback delay network, and are also approximated block by block with exponential smoothing. This allows the decay rate of the late reverberation to change gradually with scene variations, forming a complete and coherent spatial sound effect together with the early reflections rendered path-by-path. Thus, the early reflections accurately represent their direction and time delay through discrete effective sound propagation paths, while the late reverberation is statistically supplemented with a dense tail by the feedback delay network. The combination of these two approaches balances accurate orientation and computational economy.

[0067] The above is a relatively complete embodiment of the present invention. Based on this, the present invention can also have various optional embodiments. As one optional embodiment, the predetermined number of frequency bands is not limited to eight octaves. In cases where higher frequency resolution is required, a one-third octave division can be used to obtain more frequency bands, thus more finely characterizing the change in material sound absorption with frequency. In cases where computing power is more limited, the number of frequency bands can be reduced to further reduce overhead. Increasing or decreasing the number of frequency bands only affects the number of columns in the acoustic material mapping table and the number of filter banks, and does not change the above-mentioned solution and rendering process.

[0068] As another optional implementation, the preset total number of probe rays, preset upper limit of reflection order, preset lower limit of radius, preset smoothing time constant, preset fade-in duration, preset fade-out duration, preset upper limit of path quantity, preset proportion threshold, and preset upper limit of acoustic solution rate can all be flexibly adjusted according to the computing power of the target device and the application's requirements for sound quality. For example, on high-end devices, the preset total number of probe rays and the preset upper limit of reflection order can be increased to obtain denser early reflections and higher-order reflected sound, while on mobile devices, the above values ​​can be appropriately reduced to obtain lower power consumption.

[0069] As another optional implementation, in addition to specular reflection, diffuse reflection scattering can be introduced at a certain proportion at the hit point in the reflection path solution, or edge diffraction paths can be added at the edges of the geometry to more completely depict the scattering of rough surfaces and the diffraction of sound waves around the edges of obstacles; correspondingly, the traversal records generated for diffuse reflection and edge diffraction also generate path signatures according to the geometric routes they have traveled and participate in merging and smoothing, thereby further enhancing the physical realism of the sound field without changing the overall framework.

[0070] Furthermore, this invention is not limited to the aforementioned indoor hall scene with an openable metal door. For outdoor canyons, tunnels, streets between buildings, and other scenes with significant reflections and obstructions, as well as complex acoustic environments with multiple sound sources emitting sound simultaneously, the above-mentioned process of this invention can be directly applied: solving for direct and reflection paths for each sound source, distinguishing and merging the paths of each sound source using path signatures carrying sound source identifiers, and filtering and binauralizing the paths of each sound source on the audio side, can accurately present the spatial sound effects of multiple sound sources in dynamic scenes.

[0071] Corresponding to the aforementioned methods, this embodiment also provides an immersive directional sound effect simulation optimization system based on a virtual engine, including a graphics processor, a central processing unit, shared memory, an audio output device, and a memory. The graphics processor is equipped with a hardware ray tracing unit and an asynchronous computing queue, responsible for building and updating the hierarchical bounding box acceleration structure, executing acoustic ray tracing tasks, and generating acoustic state snapshots; the central processing unit runs the aforementioned audio rendering thread, completing path signature matching, state smoothing of three types of paths, and binaural rendering; the shared memory is equipped with the aforementioned dual buffers, and the acoustic state snapshots are copied from the graphics processor's video memory to the shared memory via the readback buffer and then passed to the audio rendering thread; the memory stores a computer program, which, when executed by the graphics processor and the central processing unit, implements the methods in the aforementioned embodiments; the output binaural signals are played through the audio output device. Figure 1 The data flow shown is the process of the collaborative work of the various components of the system. The values ​​of the various parameters in the system and the optional implementation methods are the same as those described in the aforementioned method, and will not be repeated here.

[0072] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An immersive directional sound effect simulation optimization method based on a virtual engine, applied to a virtual engine running a graphics rendering pipeline, wherein the graphics rendering pipeline constructs or incrementally updates the hierarchical bounding box acceleration structure of scene geometry in each graphics frame, characterized in that... Includes the following steps: Step 1: Periodically perform acoustic propagation solving at an acoustic solution rate lower than the graphics frame rate and set independently. Submit the acoustic ray tracing task to the hardware ray tracing unit of the graphics processor. The acoustic ray tracing task and the graphics ray tracing task access the same bounding box acceleration structure. Determine the acoustic parameters in a predetermined number of frequency bands based on the graphics material identifier carried by the hit triangle facets and use them to determine the frequency band energy gain. Solve to obtain the effective sound propagation path from the sound source to the listener. The path parameters of each effective sound propagation path include propagation delay, direction of arrival, and frequency band energy gain. Generate a path signature for each effective sound propagation path based on the sound source and the hit triangle facets passed through in sequence. Pack the path parameters and path signatures of all effective sound propagation paths into an acoustic state snapshot for output. Step 2: The audio rendering thread runs continuously in units of audio blocks and maintains the active path set. The audio block rate is higher than the acoustic solution rate. Whenever a new acoustic state snapshot is adopted, the path signature set of the acoustic state snapshot is matched with the path signature set of the active path set: the path whose path signature exists in both sets is a persistent path, the path whose path signature exists only in the acoustic state snapshot is a new path, and the path whose path signature exists only in the active path set is a dead path. For continuous paths, the path parameters of the continuous path are smoothed to the corresponding path parameters in the acoustic state snapshot in each audio block; for emerging paths, the crossband energy gain is increased to the corresponding crossband energy gain in the acoustic state snapshot within a preset fade-in duration; for dying paths, the crossband energy gain is attenuated to 0 within a preset fade-out duration and then the dying path is removed from the active path set. Step 3: Within each audio block, the sound source signal is rendered in a binaural manner based on the smoothed path parameters of each path in the active path set to obtain the output binaural signal.

2. The method according to claim 1, characterized in that, The hierarchical bounding box acceleration structure uses triangular facets as leaf nodes and consists of two levels: a top-level acceleration structure and a bottom-level acceleration structure, residing in the graphics processor's memory. The bottom-level acceleration structure for static geometry is built once during level loading and reused continuously in subsequent graphics frames. The bottom-level acceleration structure for dynamic geometry is refreshed in situ within graphics frames where pose or shape changes. Acoustic ray tracing tasks are submitted via an asynchronous computation queue. Acoustic ray tracing tasks and graphics ray tracing tasks belong to two independent command queues. Both have read-only access to the hierarchical bounding box acceleration structure and can be executed concurrently. Acoustic ray tracing tasks are scheduled to be executed in the graphics rendering pipeline during idle time slots after the hierarchical bounding box acceleration structure is updated in the current graphics frame and before the graphics ray tracing task in the current graphics frame starts.

3. The method according to claim 1, characterized in that, The acoustic parameters for a predetermined number of frequency bands are obtained by querying a pre-established acoustic material mapping table. The acoustic material mapping table uses graphic material identifiers as indexes and records the acoustic parameters of each graphic material identifier in a predetermined number of frequency bands. The acoustic parameters include the absorption coefficient and the transmission coefficient. The predetermined number of frequency bands consists of 8 octave bands, with the center frequency of each octave band increasing from 62.5 Hz to 8000 Hz. The air absorption coefficient is pre-configured based on ambient temperature, relative humidity, and atmospheric pressure. The air absorption factor for each frequency band is equal to the power of the natural constant, and the exponent is the negative of the product of the air absorption coefficient of the corresponding frequency band and the total length of the propagation path.

4. The method according to claim 3, characterized in that, The sound propagation solution performs direct path and reflection path calculations for each sound source. The direct path calculation involves emitting a direct detection ray from the sound source to the listener. This ray uses an arbitrary hit traversal mode to collect all hit triangles along its path. Based on the graphic material identifiers carried by the hit triangles, the acoustic material mapping table is consulted to obtain the frequency band transmission coefficients. The frequency band transmission coefficients of all hit triangles are then multiplied band by band. When the number of hit triangles is 0, the multiplication result is set to 1. The direct path constitutes a valid sound propagation path. The frequency band energy gain is equal to the product of the product of the result, the air absorption factor of the corresponding frequency band, and the free field propagation attenuation. The total propagation path length of the direct path is taken as the straight-line distance from the sound source position to the listener position. The free field propagation attenuation is equal to the square of the total propagation path length and the reciprocal of the product of pi and the value 4. The reflection path is solved as follows: the number of detection rays emitted from the sound source position along the spherical Fibonacci grid direction is equal to the total number of preset detection rays. A detection sphere is constructed with the listener position as the center and the detection sphere radius as the radius. The detection sphere radius is taken as the lower limit of the preset radius and the adaptive radius. The larger of the two values ​​is used. The adaptive radius is equal to twice the cumulative path length divided by the square root of the preset total number of probe rays. The cumulative path length is the path length that the probe ray has traveled from the sound source position. When the detection sphere radius is taken as the adaptive radius, the expected value of the number of traversal records corresponding to any mirror source is 1 and is decoupled from the propagation distance. After each probe ray hits the triangular facet, it continues to propagate in the direction of mirror reflection, experiencing at most the upper limit of the preset reflection order. The hit shader reads the graphic material identifier of the hit triangular facet and completes the acoustic material mapping table lookup. The frequency band reflection attenuation is... The intermediate values ​​of successive multiplication are stored in the ray load data and are transmitted between adjacent reflections of the probe ray. When the propagation line segment of the probe ray intersects the detection sphere after completing at least one reflection, a crossing record is generated. The crossing point is the point on the propagation line segment that is closest to the listener's position. After the record is made, the probe ray continues to propagate. Each crossing record includes the propagation delay, the direction of arrival, and the frequency band energy gain. The propagation delay is equal to the cumulative path length at the crossing point divided by the speed of sound. The direction of arrival is the reverse unit vector of the propagation direction of the probe ray at the crossing moment and is stored in the world coordinate system.

5. The method according to claim 4, characterized in that, In the frequency band energy gain of each traversal record, each frequency band component is equal to the air absorption factor of the corresponding frequency band divided by the normalized area, and then multiplied by the reflection attenuation product of the corresponding frequency band; the normalized area is equal to the product of the preset total number of detection rays, pi, and the square of the radius of the detection sphere. The product of reflection attenuation is equal to the product of the reflection retention coefficients of each reflection before crossing in the corresponding frequency band. The reflection retention coefficient is equal to the difference between the value 1 and the absorption coefficient of the triangular facet hit by the corresponding reflection. The total propagation path length corresponding to the crossing record is the cumulative path length at the crossing point.

6. The method according to claim 5, characterized in that, The path signature is composed of the source identifier of the sound source and the geometric instance identifiers and patch indices of each hit triangle in sequence, concatenated according to the reflection order. The path signature of the direct path is composed of the source identifier and a preset reserved value. For all traversal records with the same path signature, the frequency band energy gain is summed band by band. The propagation delay and the direction of arrival are taken as the weighted average of the sum of the frequency band energy gains. After the direction of arrival is weighted and averaged, it is normalized to a unit vector and merged into one effective sound propagation path. The expected value of the frequency band energy gain after merging is consistent with the energy attenuation of the corresponding mirror source under the mirror source model.

7. The method according to claim 1, characterized in that, Matching is achieved through a hash table with the path signature as the key; smoothing of the continuous path is exponential smoothing, which is: the smoothing value of each audio block is equal to the smoothing value of the previous audio block plus the product of the smoothing coefficient and the difference, and the difference is equal to the difference between the path parameter corresponding to the acoustic state snapshot and the smoothing value of the previous audio block. The smoothing coefficient is equal to the difference between the value of 1 and the power of the natural constant. The exponent is the negative of the ratio of the audio block duration to the preset smoothing time constant. The preset smoothing time constant is set separately for three types of path parameters: propagation delay, direction of arrival, and cross-band energy gain. The three components of the direction of arrival are smoothed and then re-normalized into unit vectors. Propagation delay smoothing is achieved through fractional delay lines. The reading position of the fractional delay line linearly transitions sample-by-sample within each audio block from the reading position of the previous audio block to the smoothed value of the current audio block. The continuous movement of the reading position generates a Doppler frequency shift. For newly generated audio blocks... The path and frequency-band energy gain are added to the active path set starting from a value of 0. The propagation delay and arrival direction are directly taken from the corresponding path parameters in the acoustic state snapshot starting from the first audio block. For dying paths, the propagation delay and arrival direction retain the final smoothed values. The boosting and attenuation of the frequency-band energy gain are performed according to the raised cosine envelope, which changes monotonically within the corresponding duration and has zero derivatives at the beginning and end. When the number of paths in the active path set reaches the preset path limit, the path with the smallest sum of frequency-band energy gain in the active path set is treated as a dying path before being added to the new path set. The proportion of the number of eliminated paths in each match to the total number of paths in the active path set is counted. When the proportion exceeds the preset proportion threshold, the acoustic solution rate is temporarily increased to the preset upper limit and the preset fade-in duration and preset fade-out duration are shortened simultaneously until the proportion falls back to within the preset proportion threshold, and then the acoustic solution rate, preset fade-in duration and preset fade-out duration are restored to their original values.

8. The method according to claim 5, characterized in that, The sound propagation solution also performs reverberation time estimation: the natural logarithm of the cumulative medium attenuation of all traversing records is linearly regressed with the propagation time delay to obtain the attenuation slope of each frequency band. The cumulative medium attenuation of each frequency band is equal to the product of the reflection attenuation of the corresponding frequency band and the air absorption factor of the corresponding frequency band. The reverberation time of each frequency band is the time required to extrapolate the energy attenuation to a predetermined number of decibels according to the attenuation slope of each frequency band. The predetermined number of decibels is 60. The reverberation time of each frequency band is output with the acoustic state snapshot. The audio rendering thread uses a feedback delay network to generate a parameterized late reverberation signal. The reverberation time of each frequency band of the feedback delay network is smoothed to the reverberation time of each frequency band in the acoustic state snapshot of each audio block. The parameterized late reverberation signal is superimposed on the output binaural signal.

9. The method according to claim 7, characterized in that, The registration of sound sources and listeners is executed on the main thread of the virtual engine; the acoustic state snapshot also carries a timestamp of the time of generation, which is copied from the graphics processor's video memory to shared memory via a readback buffer, and then passed to the audio rendering thread through a double buffer in shared memory. The double buffer switches its read / write pointer as a whole after the acoustic state snapshot is written, and the audio rendering thread only adopts the acoustic state snapshot with the latest timestamp; each path in the active path set records its associated sound source, and the arrival direction is stored in the world coordinate system; in step 3, for each path in the active path set, the square root of the smoothed frequency band energy gain is used as the frequency band amplitude gain, and then compared with each frequency band... A one-to-one bandpass filter bank performs frequency band filtering and weighted summation on the sound source signals of the path's sound sources. The filtering result is fed into a fractional delay line and sampled according to the smoothed propagation delay. The smoothed arrival direction is transformed to the listener's local coordinate system by transposing the rotation matrix corresponding to the listener's orientation. The listener's orientation is obtained when the listener registers and is represented by a quaternion. The head correlation transfer function is selected from the head correlation transfer function library according to the transformed direction in the nearest neighbor manner of azimuth and elevation angles. The sampling result is convolved to obtain the path binaural signal components. The path binaural signal components of all paths in the active path set are superimposed to obtain the output binaural signal.

10. An immersive directional sound effect simulation and optimization system based on a virtual engine, characterized in that, It includes a graphics processor, a central processing unit, shared memory, an audio output device, and a memory; the graphics processor is equipped with a hardware ray tracing unit and an asynchronous computing queue; the memory stores a computer program; when the computer program is executed by the graphics processor and the central processing unit, it implements the immersive directional sound effect simulation optimization method based on a virtual engine according to any one of claims 1 to 9, wherein the sound propagation solution is executed on the graphics processor, the audio rendering thread runs on the central processing unit, the acoustic state snapshot is transferred from the graphics processor to the audio rendering thread via shared memory, and the output binaural signal is played through the audio output device.

Images