Acoustic ray tracing method and device, electronic equipment, wearable equipment and storage medium

Abstract

According to the acoustic light ray tracing method and device, the electronic equipment, the wearable equipment and the storage medium provided by the invention, the collision information and the report path corresponding to the light rays emitted by the listener position are determined, and the reflection tree is constructed according to the report path; for each sound source, determining a plurality of first propagation paths from the sound source position to the listener position according to the collision information and the sound source position, and determining a plurality of second propagation paths from the sound source position to the listener position according to the reflection tree; for each sound source, determining an echogram corresponding to the first propagation path, determining an echogram corresponding to the second propagation path, determining a filter corresponding to the sound source according to the echograms, and processing an unspatialized audio stream of the corresponding sound source according to the filter to determine a reverberation signal, the constructed reflection tree being a reflection tree related to a listener, the method has the advantages of being small in calculation amount and high in accuracy, and therefore modeling of the combined path of the reflection path and the diffraction path is achieved to obtain an accurate reverberation signal.

Description

Technical Field

[0001] This invention relates to the field of audio signal processing technology, and in particular to an acoustic ray tracing method, apparatus, electronic device, wearable device, and storage medium. Background Technology

[0002] Spatial audio technology is a user-centric technique that processes sound that lacks spatial characteristics to make it sound like it possesses specific spatial features, thus making the content heard by the user more realistic. In virtual reality or mixed reality applications, spatial audio technology can match the spatial characteristics of sound with visual content, resulting in a better sense of immersion.

[0003] In virtual reality or mixed reality applications, real-time acoustic modeling enables spatial audio to match the acoustic characteristics of a defined space (virtual space or the user's real-world space). This allows the content the user hears to complement the visual content, providing a more immersive experience. For example, when the sound source changes, the content the listener hears also changes. Typically, real-time acoustic modeling uses acoustic ray tracing to model the acoustic path from the sound source to the listener based on the current spatial geometry, acoustic parameters, sound source, and listener's position. The unspatialized audio stream is then processed based on the modeling results to obtain an audio stream that matches the expected acoustic characteristics.

[0004] However, ray tracing methods equate sound propagation to particle motion, neglecting the diffraction effect of sound waves. When there is obstruction between the sound source and the listener, sound diffraction is a crucial pathway for propagation. Without modeling diffraction, the determined spatial audio will lack realism. For example, when there is obstruction between the sound source and the listener, the sound may be completely inaudible in the simulated audio, only to suddenly reappear when the listener moves to a visible location. This severely impacts the continuity and immersion of the spatial audio. Therefore, modeling the diffraction process of sound propagation to determine accurate reverberation signals is a pressing technical problem. Summary of the Invention

[0005] This invention provides an acoustic ray tracing method, apparatus, electronic device, wearable device, and storage medium for modeling the reflection, scattering, and diffraction of sound propagation to accurately determine the reverberation signal.

[0006] In a first aspect, the present invention provides an acoustic ray tracing method, comprising:

[0007] The collision information and reporting path corresponding to the light emitted from the listener's location are determined, and a reflection tree is constructed based on the reporting path; the collision information is the information of each collision point obtained by the light colliding with the scene model multiple times; the reporting path is the path composed of the reflecting surface or diffraction edge corresponding to the collision point.

[0008] For each sound source, multiple first propagation paths from the sound source location to the listener location are determined based on the collision information and the sound source location, and multiple second propagation paths from the sound source location to the listener location are determined based on the reflection tree; the second propagation path is a combination of a reflection path and a diffraction path.

[0009] For each sound source, determine the echo map corresponding to the first propagation path, and determine the echo map corresponding to the second propagation path;

[0010] The filter corresponding to the sound source is determined based on the echo map, and the unspatialized audio stream of the corresponding sound source is processed according to the filter corresponding to each sound source to determine the reverberation signal.

[0011] Optionally, determining the collision information and reporting path corresponding to the light emitted from the listener's location, and constructing a reflection tree based on the reporting path, includes: determining the collision information and reporting path corresponding to the light emitted from the listener's location, and constructing a reflection tree based on the reporting path when at least one of the following conditions is met: the listener's location changes, the scene model changes, or the reflectivity of any reflective surface in the scene model changes.

[0012] Optionally, the scene model is represented using triangulation; the method further includes:

[0013] Determine the diffraction edge in the scene model; the diffraction edge is the coincident side of two non-coplanar triangles;

[0014] Determine and save diffraction edge information; the diffraction edge information corresponds to the diffraction edge; wherein, when the dynamic geometry in the scene model changes, the corresponding diffraction edge information is updated;

[0015] Accordingly, determining the collision information and reporting path corresponding to the light emitted from the listener's location includes: performing ray tracing on the light emitted from the listener's location, determining the collision information, and determining the reporting path based on the diffraction edge information.

[0016] Optionally, the reporting path includes a reflection path and a diffraction path; ray tracing is performed on the light emitted from the listener's position to determine collision information, and the reporting path is determined based on the diffraction edge information, including:

[0017] For each ray of light, repeat the following steps to determine the collision information each time the ray collides, until the condition for ending the tracking of the ray is met:

[0018] When the light ray collides with the scene model, the collision information for this collision is determined;

[0019] When it is determined that the tracking of the ray will not end, the reflection path is determined and the direction of the reflected ray is calculated to determine whether a collision will occur with another reflective surface in the scene model based on the direction of the reflected ray.

[0020] Furthermore, when it is determined that the tracking of the light ray will not end, it is determined whether diffraction will occur based on the diffraction edge information. When diffraction occurs, the diffraction path is determined, and the direction of the diffracted light ray is calculated to determine whether it will collide with another reflective surface in the scene model based on the direction of the diffracted light ray.

[0021] Optionally, determining whether diffraction will occur based on the diffraction edge information includes:

[0022] Determine the point of collision between the light ray and the reflecting surface;

[0023] When the collision point is close to the edge of the triangle corresponding to the reflecting surface, it is determined whether the edge is the saved diffraction edge; the collision point is determined to be close to the edge of the triangle corresponding to the reflecting surface based on the transformed centroid coordinates of the collision point.

[0024] If the edge is one of a plurality of diffraction edges that are preserved, then two shadow regions are determined according to the two reflecting surfaces corresponding to the diffraction edge. When the light is in either of the shadow regions, diffraction is determined to occur.

[0025] Optionally, a reflection tree is constructed based on the reported path, including:

[0026] The reported path is deduplicated to obtain a deduplicated path; the reported path includes a start node and an end node; the start node or the end node is a reflecting surface or a diffraction edge;

[0027] The listener is identified as the root node of the reflection tree, and the following steps are repeated until the maximum depth of the reflection tree reaches a preset depth:

[0028] For each node in the reflection tree, find the target node that the node can connect to from all the deduplicated paths, and determine the target node as the child node of the node.

[0029] Optionally, the method further includes:

[0030] When determining the reflection tree, if the node is the root node, the spatial position corresponding to the root node is determined as the listener's position;

[0031] When the node is a reflective surface, determine the mirror position of the spatial position of the parent node of the node relative to the reflective surface, and determine the mirror position as the spatial position of the node.

[0032] When the node is a diffraction edge, the center position of the diffraction edge is determined as the diffraction point or the spatial position corresponding to the node.

[0033] Optionally, the reflection path in the second propagation path is a specular reflection path; multiple second propagation paths from the sound source location to the listener location are determined based on the reflection tree, including:

[0034] For each sound source, determine the reflecting surface and diffraction edge where the light emitted from the sound source location collides with the scene model for the first time, so as to obtain a list of visible nodes of the sound source.

[0035] For each leaf node in the reflection tree, when the leaf node is in the sound source visible list, it is determined whether the propagation path corresponding to the leaf node is a valid path;

[0036] When the leaf node is not in the visible list of sound sources, the propagation path corresponding to the leaf node is determined to be an invalid path;

[0037] The second propagation path is determined based on the established legal path; the second propagation path is a path obtained by sequentially connecting the sound source location, the reflection points or diffraction points corresponding to each node in the legal path, and the listener's location.

[0038] Optionally, determining whether the propagation path corresponding to the leaf node is a valid path includes:

[0039] If every child node from the leaf node to the root node satisfies the target condition, then the path from the leaf node to the root node is determined to be a valid path; otherwise, it is an invalid path.

[0040] When the node is a reflective surface, the target condition is: the reflection point corresponding to the reflective surface is within the triangle corresponding to the reflective surface and the corresponding path is not obstructed; the position of the reflection point is related to the spatial position corresponding to the reflective surface and the spatial position of the next level node.

[0041] When the node is a diffraction edge, the target conditions are: the corresponding path is not occluded, and the preceding and following nodes of the node are located in two different shadow areas corresponding to the node.

[0042] Optionally, determining the echo map corresponding to the second propagation path includes:

[0043] For the sound source, determine the initial energy corresponding to the sound source;

[0044] For each second propagation path, the distance attenuation coefficient and air absorption coefficient are determined based on the total length of the second propagation path. The reflection attenuation coefficient of each reflecting surface in the second propagation path is determined. The reflection attenuation coefficients of each reflecting surface are multiplied to obtain the overall reflection attenuation coefficient. The diffraction coefficient is determined based on the diffraction edge information of each diffraction edge in the second propagation path. The reflection attenuation coefficient of the reflecting surface is related to the reflectivity and scattering rate of the reflecting surface.

[0045] The multiplication result of the initial energy, the distance attenuation coefficient, the air absorption coefficient, the overall reflection attenuation coefficient, and the diffraction coefficient is determined as the energy received by the listener;

[0046] The direction in which the last node points to the listener's position is defined as the receiving direction;

[0047] The energy received by the listener, the receiving direction, and the propagation time corresponding to the second propagation path are determined as the echo map corresponding to the second propagation path.

[0048] In a second aspect, the present invention provides an acoustic ray tracing device, comprising:

[0049] The listener tracking module is used to determine the collision information and reporting path of the light emitted from the listener's location; the collision information is the information of each collision point obtained by the light colliding with the scene model multiple times;

[0050] A reflection tree construction module is used to construct a reflection tree based on the reported path; the reported path is a path composed of the reflecting surface or diffraction edge corresponding to the collision point.

[0051] The sound source tracking module is used to determine, for each sound source, multiple first propagation paths from the sound source location to the listener location based on the collision information and the sound source location;

[0052] The path search module is used to determine multiple second propagation paths from the sound source location to the listener location based on the reflection tree; the second propagation path is a combination path of specular reflection path and diffraction path;

[0053] The sound source tracking module is also used to determine the echo map corresponding to the first propagation path for each sound source.

[0054] The path modeling module is used to determine the echo map corresponding to the second propagation path;

[0055] A filter synthesis module is used to determine the filter corresponding to the sound source based on the echo map;

[0056] The processing module is used to process the unspatialized audio stream of each sound source according to the filter corresponding to each sound source to determine the reverberation signal.

[0057] Thirdly, the present invention provides an electronic device, comprising: at least one processor and a memory;

[0058] The memory stores computer-executed instructions;

[0059] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the method as described in any of the first aspects.

[0060] Fourthly, the present invention provides a wearable device including a processing unit; the processing unit is configured to perform the method as described in any of the first aspects.

[0061] Fifthly, the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the method of any one of the first aspects.

[0062] In a sixth aspect, the present invention provides a computer program product comprising a computer program that, when executed by a processor, implements the method described in any of the first aspects.

[0063] This invention provides an acoustic ray tracing method, apparatus, electronic device, wearable device, and storage medium. It determines the collision information and reporting path corresponding to the light emitted from the listener's location, and constructs a reflection tree based on the reporting path. The collision information consists of information about each collision point obtained from multiple collisions between the light and the scene model. The reporting path is a path composed of reflecting surfaces or diffraction edges corresponding to the collision points. For each sound source, multiple first propagation paths from the sound source location to the listener's location are determined based on the collision information and the sound source location. Multiple second propagation paths from the sound source location to the listener's location are determined based on the reflection tree. The second propagation path is a combination of a reflection path and a diffraction path. For each sound source, an echo map corresponding to the first propagation path and an echo map corresponding to the second propagation path are determined. The unspatialized audio stream of the corresponding sound source is processed based on the determined echo maps to determine the reverberation signal. The constructed reflection tree is a listener-related reflection tree, which has the advantages of low computational cost and high accuracy, thereby achieving modeling of the combination path of reflection and diffraction paths in the sound propagation process to obtain an accurate reverberation signal. Attached Figure Description

[0064] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0065] Figure 1 An application scenario diagram provided by an embodiment of the present invention;

[0066] Figure 2 An outdoor diffraction diagram provided for an embodiment of the present invention;

[0067] Figure 3 A schematic flowchart of an acoustic ray tracing method provided in an embodiment of the present invention;

[0068] Figure 4 This is a schematic diagram of an overall architecture for generating spatial audio provided in an embodiment of the present invention;

[0069] Figure 5 A detailed schematic diagram of an architecture for generating spatial audio is provided for an embodiment of the present invention;

[0070] Figure 6 A schematic diagram of a diffraction edge provided for an embodiment of the present invention;

[0071] Figure 7 This is a schematic diagram illustrating the specific process of a listener tracking module provided in an embodiment of the present invention;

[0072] Figure 8 A schematic diagram illustrating a method for calculating the reflection direction provided in an embodiment of the present invention;

[0073] Figure 9 A schematic diagram of random diffraction provided in an embodiment of the present invention;

[0074] Figure 10 A top view schematic diagram of random diffraction provided in an embodiment of the present invention;

[0075] Figure 11 A schematic diagram of a path report provided in an embodiment of the present invention;

[0076] Figure 12 This is a schematic flowchart of a sound source tracking module provided in an embodiment of the present invention;

[0077] Figure 13 This is a schematic diagram of a specular reflection path provided in an embodiment of the present invention;

[0078] Figure 14 A schematic diagram of a scattering path provided in an embodiment of the present invention;

[0079] Figure 15 A schematic diagram of a reflection tree provided in an embodiment of the present invention;

[0080] Figure 16 A schematic diagram of a mirror position and propagation path provided for an embodiment of the present invention;

[0081] Figure 17 This is a schematic diagram illustrating the specific process of a path search module provided in an embodiment of the present invention;

[0082] Figure 18 A schematic diagram of a diffraction path modeling provided in an embodiment of the present invention;

[0083] Figure 19 A schematic diagram of an echo map provided in an embodiment of the present invention;

[0084] Figure 20 This is a schematic diagram of the structure of a filter synthesis module provided in an embodiment of the present invention;

[0085] Figure 21 An example diagram for reconstructing a frequency band impulse response from an echo map, provided as an embodiment of the present invention;

[0086] Figure 22 This is a schematic diagram of the structure of an acoustic ray tracing device provided in an embodiment of the present invention;

[0087] Figure 23 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention.

[0088] The accompanying drawings have illustrated specific embodiments of the invention, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the invention in any way, but rather to illustrate the concept of the invention to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0089] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention.

[0090] In virtual reality or mixed reality applications, spatial audio technology can be used to match the spatial characteristics of sound with visual content to achieve a better sense of immersion. Figure 1 An application scenario diagram provided by an embodiment of the present invention, such as... Figure 1As shown, when there are two sound sources in a space, the listener can simultaneously receive the spatial audio of these two sound sources in the virtual or real space. In other words, the spatial audio ultimately received by the listener is related to the location of the sound source, the listener's location, and the geometry of the room, thus enabling the user to combine the spatial audio they hear with the visual content to provide a stronger sense of realism.

[0091] A common method for determining spatial audio is to use acoustic ray tracing to model the acoustic path from the sound source to the listener based on the current spatial geometry, acoustic parameters (such as the reflectivity of reflective surfaces in a room), the location of the sound source, and the location of the listener. Here, modeling refers to determining filters with propagation characteristics from the sound source to the listener based on the energy received by the listener, so as to process the unspatialized audio stream according to the filters to obtain spatial audio that matches the expected acoustic characteristics.

[0092] The basic process of acoustic ray tracing is as follows: energy-carrying rays are randomly emitted from the sound source location, and the propagation of each ray in the room is tracked. When the ray collides with the wall of the room, the energy it carries is attenuated, and then the ray is reflected and continues to propagate. This process is repeated until the distance the ray has traveled reaches a preset upper limit, or the remaining energy reaches a preset lower limit. For each collision point, if there is no obstruction between it and the listener, the energy scattered to the listener from that collision point is calculated. All the energy scattered to the listener is statistically analyzed to generate an acoustic impulse response. Based on this acoustic impulse response, the unspatialized audio signal can be processed to obtain the reverberation signal received by the listener.

[0093] However, the aforementioned ray tracing method has a problem: ray tracing equates the propagation of sound to the motion of particles, neglecting the diffraction effect of sound waves. Diffraction refers to the phenomenon where sound waves can bypass obstacles and continue propagating. In the real world, when there is an obstruction between the sound source and the listener, sound wave diffraction is a crucial pathway for sound propagation between them, thus enabling the perception of "hearing the sound before seeing the person." Without modeling diffraction, the spatial audio simulated using ray tracing algorithms will lack realism.

[0094] For scenarios with weak reflections, such as Figure 2 In the outdoor scenario shown, there is obstruction between the sound source and the listener. Diffraction often becomes the primary path for sound wave propagation. Without modeling diffraction, the simulated audio might be completely inaudible when there is obstruction between the sound source and the listener. Conversely, the sound might suddenly reappear when the user or the sound source moves, severely impacting the continuity and immersion of the spatial audio. Therefore, modeling diffraction is crucial in the above scenario.

[0095] One approach to diffraction modeling is to solve the wave equation to model the diffraction of sound waves more accurately. However, for virtual scenes with dynamic interaction and large scale, the computational load required by the above method is enormous, and the running time is difficult to meet the requirements of real-time interaction.

[0096] Another diffraction modeling method involves pre-analyzing the static scene, exhaustively enumerating all possible intermediate diffraction paths from one diffraction edge to another, and calculating the filter corresponding to each path, storing the results in a diffraction path provider. During the rendering phase, based on the positions of the sound source and the listener, the visible diffraction edges of each are located, and all possible diffraction paths are searched from the diffraction path provider to construct a complete diffraction path in the form of "from the sound source through any number of diffraction edges to the listener." The filter corresponding to the complete diffraction path is then calculated, thereby processing the unspatialized audio stream to obtain a simulated diffraction signal.

[0097] The above method saves a significant amount of path-finding work during the rendering process by pre-calculating and generating diffraction path providers. However, it still has some drawbacks: First, it can only pre-analyze and generate diffraction path providers for static parts of the scene. When dynamic geometry exists in the scene, it needs to be re-analyzed when the dynamic geometry moves, rotates, scales up, or shrinks. This analysis process requires a large amount of computation, thus reducing modeling speed. Second, in some scenes, sound waves may need to travel through a combination of diffraction and reflection paths to reach the listener, such as... Figure 2 In the scenario shown, sound waves travel from the sound source to the listener via a combination of diffraction-reflection-diffraction paths, but the method described above cannot support modeling such combinations. Third, in some complex scenarios, the number of diffraction edges can be very large, while the number of diffraction paths increases exponentially relative to the number of diffraction edges, making it impractical to exhaustively enumerate all possible diffraction paths. Therefore, a new diffraction modeling method is needed to address these issues.

[0098] Furthermore, ray tracing involves tracking a large number of rays, each of which reflects multiple times within a room, resulting in a significant computational burden. To address this, reverse ray tracing is employed, splitting a complete path into two parts for separate tracking: listener tracking and sound source tracking. This decouples the listener from the sound source. Therefore, when the sound source moves while the listener remains stationary, the tracking results of the first part remain unchanged, allowing the reuse of previous listener tracking results. The listener tracking portion can be omitted, requiring only the sound source tracking portion to run, thus effectively reducing the computational load.

[0099] To address the aforementioned issues, this application considers that modeling diffraction paths should ideally be easily integrated with ray tracing methods. This would generate modeling results that simultaneously incorporate effects such as reflection, scattering, and diffraction, thereby enabling unified processing of unspatialized audio. Therefore, based on the scattering path determined by inverse ray tracing from the sound source to the listener, this application obtains a path related to the listener's position, consisting of reflecting surfaces and diffraction edges, based on the results of inverse ray tracing. This results in a reflection tree, which can be understood as a connectivity model between all reflecting surfaces and diffraction edges related to the listener's current position. This model is independent of the sound source, thus allowing the generation of a combined path of reflection and diffraction paths based on the reflection tree and the sound source position.

[0100] By constructing a reflection tree based on the results of inverse ray tracing, a reflection tree that is only related to the listener's position can be obtained, resulting in a smaller reflection tree and lower computational cost. Furthermore, since the reflection tree includes reflective surfaces, a combined path of reflection and diffraction can be obtained. When the scene model changes, i.e., when dynamic geometry exists, inverse ray tracing is performed again, thus updating the constructed reflection tree simultaneously. In other words, constructing a reflection tree based on inverse ray tracing results does not require additional computation, as modeling scattering paths based on inverse ray tracing is an essential part of the system. Moreover, when dynamic geometry exists in the scene, since inverse ray tracing is performed again, there is no need to re-analyze and construct the reflection tree. In summary, inverse ray tracing can be used to model scattering paths, and the reflection tree constructed based on the results of inverse ray tracing can model a combined path of reflection and diffraction without analyzing the scene model to determine the diffraction path, offering advantages such as lower computational cost and higher accuracy throughout the modeling process.

[0101] Figure 3 This is a flowchart illustrating an acoustic ray tracing method provided in an embodiment of the present invention. The method includes steps S301 to S303:

[0102] Step S301: Determine the collision information and reporting path corresponding to the light emitted from the listener's position, and construct a reflection tree based on the reporting path; the collision information is the information of each collision point obtained by the light colliding with the scene model multiple times; the reporting path is the path composed of the reflecting surface or diffraction edge corresponding to the collision point.

[0103] The ray tracing in this application is inverse ray tracing, which means tracing the light rays emitted from the listener's position. By performing the inverse tracing process, ray tracing can be divided into two stages, thereby decoupling the listener from the sound source. It should be noted that the light rays emitted from the listener's position here do not refer to real light rays, but can be simulated light rays, such as multiple light rays emitted from the listener's position represented by equations. The ray tracing here is used to simulate the propagation of sound.

[0104] The scene in which the user and listener are situated can be represented by a scene model. Optionally, the scene can be a room, in which case the scene model can be used to indicate the size of the room, the position of each object in the room, and / or the reflective surfaces contained in the room and the reflective surfaces of each object, etc.

[0105] Optionally, multiple random light rays can be emitted from the listener's position. These rays will collide with the scene model during propagation, randomly reflecting and scattering. The collision information can be determined using the first-stage tracking process (i.e., listener tracking). This collision information can involve tracking each light ray emitted from the listener's position and recording the collision point when it collides with the scene model. For example, the collision information can include the location of the collision point and the reflective surface it occupies.

[0106] For example, three rays are emitted from the listener's position. For each ray, the corresponding collision information can be determined. For one of the rays, it may collide with reflective surface 1 (creating collision point 1), reflective surface 3 (creating collision point 2), and reflective surface 2 (creating collision point 3) in the scene model in sequence. After the condition for ending ray tracing is met, the ray tracing ends. Here, collision point 3 is the last collision point corresponding to the ray, so the information of these three collision points can be determined.

[0107] Tracing the light rays emitted from the listener's position to the last collision point reveals the propagation path from the listener's position to the last collision point, independent of the sound source's location. In other words, regardless of the number or location of the sound sources, the collision information of each light ray remains unchanged. Therefore, the path of the light rays from the listener's position to the last collision point can be traced; this tracing process can be considered listener tracking.

[0108] During the listener tracking process, when each ray is tracked, if the ray collides with the reflecting surface, it is determined whether there is a diffraction edge based on the collision result. The reporting path is then determined based on the reflecting surface or diffraction edge corresponding to the collision point. A reflection tree is then constructed based on the reporting path. The path from the leaf node to the root node in the reflection tree is a possible propagation path of the sound.

[0109] Step S302: For each sound source, determine multiple first propagation paths from the sound source location to the listener location based on the collision information and the sound source location, and determine multiple second propagation paths from the sound source location to the listener location based on the reflection tree; the second propagation path is a combination path of reflection path and diffraction path.

[0110] After determining the collision information, ray tracing can continue based on the collision information. Ray tracing here is to trace the path from each collision point to the sound source location. This path is the first propagation path, which is related to the sound source location. When the sound source location is different, the first propagation path is different. This tracing process can be regarded as sound source tracing.

[0111] For a given sound source, based on the location of each collision point, the path from each collision point to the sound source can be determined. Then, based on the path from the listener's location to each collision point, multiple complete propagation paths from the listener to the sound source can be obtained, and it can be equivalently assumed that there is a common propagation path from the sound source location to the listener's location.

[0112] For example, when a ray of light collides sequentially with reflective surface 1 (creating collision point 1), reflective surface 3 (creating collision point 2), and reflective surface 2 (creating collision point 3) in the scene model, there are three propagation paths from the sound source position to the listener position. Path 1 is the ray of light from the sound source position through collision point 3 to the listener position; path 2 is the ray of light from the sound source position through collision point 3 and collision point 2 to the listener position; and path 3 is the ray of light from the sound source position through collision point 3, collision point 2, and collision point 1 to the listener position.

[0113] The paths determined above based on the locations of various collision points and sound sources are mostly scattering paths, meaning that the specular reflection of the incident light rays fails to pass through the sound source. Therefore, a ray of light scattered to the sound source is calculated. However, the specular reflection path is a real-world path, and the energy of the specular reflection light rays is relatively high, significantly impacting the listener's perception. Therefore, based on the reflection tree and the sound source location, a combined path of specular reflection and diffraction can be determined from the sound source to the listener, which is the second propagation path.

[0114] Step S303: For each sound source, determine the echo map corresponding to the first propagation path, and determine the echo map corresponding to the second propagation path;

[0115] Step S304: Determine the filter corresponding to the sound source based on the echo map, and process the unspatialized audio stream of the corresponding sound source according to the filter corresponding to each sound source to determine the reverberation signal.

[0116] For each sound source, the echo map corresponding to the first propagation path and the echo map corresponding to the second propagation path can be determined. The echo map represents the arrival time, direction of arrival, and energy received by the listener for each propagation path from the sound source location to the listener location. For example, if there are three propagation paths from the sound source to the listener, the echo map corresponding to each path can be determined.

[0117] Based on the echo maps corresponding to each sound source, the filter corresponding to that sound source can be determined. Optionally, the filter can be in the form of an Ambisonics filter bank. After generating the filter corresponding to each sound source, a fast convolution method can be used to convolve the unspatialized audio stream of the corresponding sound source based on the filter to obtain the Ambisonics reverberation signal.

[0118] This invention proposes an acoustic ray tracing method. First, random diffraction is incorporated into the inverse ray tracing method, and the results of random ray tracing are statistically analyzed. A reflection tree is constructed based on all traced reflection and diffraction nodes. All legal combinations of reflection and diffraction paths are found based on the reflection tree. Finally, all found paths are modeled to obtain a filter. This approach has the following advantages: it can model combinations of reflection and diffraction paths; it does not require prior calculation and analysis of diffraction paths, thus enabling real-time modeling of scenes containing dynamic geometry; it does not require exhaustively enumerating all diffraction edges and paths in the scene, but instead utilizes the ray tracing results to construct and traverse only the traced portions of the reflection tree, reducing computational and storage requirements for complex scenes; the diffraction modeling results are merged with the filters generated by random ray tracing, allowing for unified processing of unspatialized audio streams during the processing stage.

[0119] Figure 4 This is a schematic diagram of an overall architecture for generating spatial audio provided in an embodiment of the present invention, such as... Figure 4As shown, the room geometry is considered the scene model in this application. The method of this application can be applied to a processing device, which includes an acoustic ray tracing module and a spatial audio rendering module. The room acoustic parameters, room geometry (also known as the scene model), sound source positions, and listener positions can be input into the processing device. The acoustic ray tracing module can process the input information to obtain filters corresponding to each sound source (including sound diffraction, scattering, and reflection effects, etc.). The obtained filters corresponding to each sound source, the unspatialized audio stream in the memory, and the listener orientation information provided by the inertial sensor are input into the spatial audio rendering module, so that the spatial audio rendering module can output a spatialized audio stream and output it to a sound playback device, such as a speaker or headphones. The process of determining the filter based on the echo map in steps S301 to S303 of this application is the execution content of the acoustic ray tracing module, and the process of determining the reverberation signal based on the filter in step S303 is the execution content of the spatial audio rendering module.

[0120] This invention provides an acoustic ray tracing method, apparatus, electronic device, wearable device, and storage medium. It determines the collision information and reporting path corresponding to the light emitted from the listener's location, and constructs a reflection tree based on the reporting path. The collision information consists of information about each collision point obtained from multiple collisions between the light and the scene model. The reporting path is a path formed by the reflecting surfaces or diffraction edges corresponding to the collision points. For each sound source, multiple first propagation paths from the sound source location to the listener's location are determined based on the collision information and the sound source location. Multiple propagation paths from the sound source location to the listener's location are determined based on the reflection tree. The second propagation path is a combination of reflection and diffraction paths. For each sound source, the echo map corresponding to the first propagation path is determined, and the echo map corresponding to the second propagation path is also determined. A filter corresponding to the sound source is determined based on the echo map. The unspatialized audio stream of each sound source is processed according to the filter corresponding to each sound source to determine the reverberation signal. The reflection tree constructed here is a listener-related reflection tree, which has the advantages of low computational cost and high accuracy. This enables the modeling of the combination of reflection and diffraction paths in the sound propagation process to obtain an accurate reverberation signal.

[0121] Optionally, determine the collision information and reporting path corresponding to the light emitted from the listener's location, and construct a reflection tree based on the reporting path, including:

[0122] When at least one of the following conditions is met, the collision information and reporting path corresponding to the light emitted from the listener's location are determined, and a reflection tree is constructed based on the reporting path:

[0123] The listener's position changes, the scene model changes, and the reflectivity of any reflective surface in the scene model changes.

[0124] The collision information corresponding to the light emitted from the listener's position is determined by the listener tracking process; however, listener tracking and reflection tree construction are not performed every time an update occurs. Optionally, listener tracking and reflection tree construction can be determined based on whether the listener's position changes, whether the scene model changes, or whether the reflectivity of any reflective surface in the scene model changes.

[0125] When the listener's position changes, the collision information when the light ray collides with the scene model will change. Even if the listener's position remains the same, the collision information will still change when the scene model changes. When the reflectivity of any reflective surface in the scene model changes, the remaining energy of the light ray after a collision can be altered, thus determining whether to continue tracking the ray; therefore, the collision information will also change.

[0126] If any one of the three conditions above is met, the listener tracking process and reflection tree construction are executed. Conversely, if none of the three conditions are met, the listener tracking process and reflection tree construction are not run, and the output results of the previous listener tracking and reflection tree construction can be reused directly.

[0127] By determining whether the listener's position, scene model, or the reflectivity of any reflective surface has changed, it is possible to accurately determine whether to perform the listener tracking process and reflectance tree construction. In some scenarios, listener tracking and reflectance tree construction can be omitted, thereby reducing the computational load.

[0128] Figure 5 This is a detailed architectural diagram illustrating a method for generating spatial audio according to an embodiment of the present invention, such as... Figure 5As shown, the entire implementation scheme is divided into three stages: preprocessing stage 110, simulation stage 120, and processing stage 130. Preprocessing stage 110 runs during system initialization, primarily aiming to find all diffraction edges in the scene and save this information for use in the simulation stage. Simulation stage 120, based on the room's scene model (3D Mesh), the diffraction edge information generated in the preprocessing stage, the acoustic reflectivity data of each reflective surface in the corresponding scene model, and the listener's and sound source's positions, models the room's acoustic characteristics using ray tracing. The modeling result is the Ambisonics filter bank corresponding to each sound source. Simultaneously, in the simulation stage, a reflection tree is constructed using the ray tracing results. Then, for each sound source, the reflection tree is traversed to search for all legal diffraction and reflection combination paths. The searched paths are modeled and accumulated into the Ambisonics filter bank generated by the ray tracing method. In processing stage 130, the estimated Ambisonics filter bank is used to process the unspatialized audio to obtain the Ambisonics reverberation signal. The room reverberation signal is added to the direct sound signal processed by HRTF (Head-Related Transfer Function) and then output to an audio playback device, such as a speaker or headphones.

[0129] Depending on the specific scenario, the simulation phase 120 and the processing phase 130 can have different update frequencies. The processing phase 130 is typically designed to generate a real-time audio stream. For example, when the sampling rate is 48000Hz and 1024 samples are processed each time, the processing phase needs to run at least once every 21.3ms (1024 / 48000*1000) to generate the real-time audio stream. The update frequency of the simulation phase 120 is set according to the complexity of the actual scenario and the system's computing power; for example, it can be set to update once every 100ms. Depending on the system architecture, the simulation phase and the processing phase can also run on different processing devices; this invention does not impose any limitations on this.

[0130] The simulation phase 120 mainly comprises six steps: listener tracking module 121, reflection tree construction module 122, sound source tracking module 123, path search module 124, path modeling module 125, and filter synthesis module 126. Among these, the listener tracking module 121 and reflection tree construction module 122 are only related to the listener's position; therefore, they only need to run when the listener's position changes, the scene model changes, or the reflectivity of the reflecting surface changes. Otherwise, this step can be skipped, and the output of the previous listener tracking module 121 and reflection tree construction module 122 can be reused to directly run the sound source tracking module 123 and path search module 124. The sound source tracking module 123, path search module 124, and path modeling module 125 run separately for each sound source, depending on both the sound source's position and the output of the listener tracking module and reflection tree construction module. Similarly, the filter synthesis module 126 runs separately for each sound source, synthesizing the output of the corresponding sound source tracking module 123 and path modeling module 125 into an Ambisonic filter bank for that sound source.

[0131] Processing stage 130 mainly includes steps such as a fast convolution module 131, an HRTF processing module 132, an Ambisonics rotation module 133, and an Ambisonics decoding module 134. The fast convolution module 131 uses a fast convolution method to convolve the Ambisonics filter bank with the input unspatialized audio signal to obtain an Ambisonics reverberation signal. The Ambisonics rotation module 133 spatially rotates the Ambisonics reverberation signal based on user orientation information provided by an inertial sensor to match the user's current orientation. The Ambisonics decoding module 134 performs binaural decoding on the rotated Ambisonics reverberation signal to obtain a binaural reverberation signal. This signal is mixed with the binaural direct sound signal output by the HRTF processing module 132 to obtain the final binaural signal used for playback.

[0132] The detailed execution process of each of the above stages is explained below.

[0133] Optionally, the scene model is represented using triangulation; the method further includes:

[0134] Determine the diffraction edge in the scene model; the diffraction edge is the coincident side of two non-coplanar triangles;

[0135] Determine and save diffraction edge information; the diffraction edge information corresponds to the diffraction edge; wherein, when the dynamic geometry in the scene model changes, the corresponding diffraction edge information is updated;

[0136] Accordingly, the collision information and reporting path corresponding to the light emitted from the listener's location are determined, including:

[0137] Ray tracing is performed on the light emitted from the listener's position to determine collision information, and the reporting path is determined based on the diffraction edge information.

[0138] When constructing a reflection tree based on the listener tracking results, diffraction edge information is also needed. Diffraction edge information is the result of analyzing static or dynamic geometry in the scene. The information corresponding to the possible diffraction edges in the scene model is stored to accurately determine the reporting path.

[0139] Optionally, the diffraction edge finding module 111 analyzes the input scene model (3D Mesh) represented by triangulation, finds and records all possible diffraction edges, and saves them in the diffraction edge data module 112. Objects in the scene model can include static and dynamic geometry. Since the movement, rotation, scaling, or reduction of dynamic geometry does not affect the existence of diffraction edges, a unified search strategy can be used for both static and dynamic geometry in the scene.

[0140] Figure 6 This is a schematic diagram of a diffraction edge provided in an embodiment of the present invention. When any two triangles in a scene satisfy the condition that the two triangles have one and only one overlapping edge, and the two triangles are not coplanar, then a diffraction edge is considered to exist, and the determined diffraction edge is this overlapping edge. For example... Figure 6 As shown, triangles 1 and 2 represent the reflecting surfaces, and at this point, diffraction edges exist.

[0141] The data recorded in the diffraction edge data module 112 includes diffraction edge information corresponding to each diffraction edge. Optionally, the diffraction edge information may include vertex index, corresponding triangle index, triangle normal vector, diffraction surface vector, diffraction edge length, diffraction angle, etc.

[0142] For static geometry, the diffraction edge information remains unchanged throughout the system's operation. However, for dynamic geometry, when it rotates, translates, enlarges, or shrinks, its triangle normal vector, diffraction surface vector, diffraction edge length, and diffraction angle need to be updated accordingly.

[0143] like Figure 5 As shown, the determined diffraction edge information can be transmitted to the listener tracking module 121 and the reflection tree construction module 122 (actually transmitted to the path modeling module 125 for use in determining the diffraction coefficients). The listener tracking module 121 needs to use the diffraction edge information when determining the reporting path to determine whether diffraction has occurred, thereby determining the reporting path.

[0144] By determining the diffraction edge information in the scene model during system initialization, the listener tracking module can accurately determine the reporting path, thereby improving the accuracy of the subsequently determined diffraction path.

[0145] Optionally, the reporting path includes a reflection path and a diffraction path; ray tracing is performed on the light emitted from the listener's position to determine collision information, and the reporting path is determined based on the diffraction edge information, including:

[0146] For each ray of light, repeat the following steps to determine the collision information each time the ray collides, until the condition for ending the tracking of the ray is met:

[0147] When the light ray collides with the scene model, the collision information for this collision is determined;

[0148] When it is determined that the tracking of the ray will not end, the reflection path is determined and the direction of the reflected ray is calculated to determine whether a collision will occur with another reflective surface in the scene model based on the direction of the reflected ray.

[0149] Furthermore, when it is determined that the tracking of the light ray will not end, it is determined whether diffraction will occur based on the diffraction edge information. When diffraction occurs, the diffraction path is determined, and the direction of the diffracted light ray is calculated to determine whether it will collide with another reflective surface in the scene model based on the direction of the diffracted light ray.

[0150] Figure 7 This is a schematic diagram illustrating a specific process of a listener tracking module provided in an embodiment of the present invention, such as... Figure 7 As shown, during ray tracing, rays emitted from the listener's position are tracked. These rays can collide with the scene model. When a ray hits a reflective surface, it undergoes reflection and scattering, depending on the reflectivity α of the surface. s The remaining energy α after reflection can be calculated. s E represents the energy currently carried by the ray. The distance between the current collision point and the previous collision point is calculated and summed to obtain the ray's propagation distance (the distance from the listener's position to the current collision point). The number of collisions, the ray's propagation distance, and the remaining ray energy determine whether to terminate the tracking of the current ray.

[0151] If the tracing termination condition is not met, determine the reflection path and calculate the direction of the reflected ray based on the scattering rate of the reflecting surface, then continue calculating the next collision point between the reflected ray and the scene model. If the tracing termination condition is still not met, it is also necessary to determine whether diffraction will occur. If diffraction is determined, the diffraction path can be determined and the direction after diffraction calculated. Repeat the above steps for all rays until all rays meet the tracing termination condition.

[0152] When the number of collisions reaches the upper limit, the propagation distance of the light reaches the upper limit, or the remaining light energy reaches the lower limit, the conditions for ending the tracking of the current light are determined to be met.

[0153] The collision information for each collision can include: the reflecting surface where the collision point is located, the specific collision location, the angle of the incident ray and the angle of the reflected ray, the accumulated distance, etc.; when diffraction occurs after the collision, the collision information can also include the diffraction edge and diffraction point, the angle of the diffracted ray, etc. The above collision information can be transmitted to the sound source tracking module 123.

[0154] Figure 8 This is a schematic diagram of a method for calculating the reflection direction provided in an embodiment of the present invention, as shown below. Figure 8 As shown, the direction of the reflected light is obtained by combining the specular reflection direction and the random scattering direction. If the incident direction is d... incident If the normal vector of the collision surface is n, then the reflection direction of the mirror is d. specular for:

[0155] d specular =d incident -2*d incident ·n

[0156] Optional, random scattering direction d scattering Let n be a randomly generated direction vector that follows a Lambert distribution relative to the normal vector n.

[0157] According to the scattering rate α of the collision surface scattering The direction of the reflected light is obtained and normalized, as shown in the following formula:

[0158] d reflection =α scattering d scattering +(1-α scattering )d specular

[0159]

[0160] When diffraction occurs, the direction of the diffracted light rays can be any random direction that meets certain conditions. These conditions are: the shadow region of the diffracted light ray is different from the shadow region of the incident light ray.

[0161] By determining whether diffraction will occur during listener tracking, the diffraction path can be accurately obtained, thereby enabling the accurate construction of the reflection tree.

[0162] Optionally, determining whether diffraction will occur based on the diffraction edge information includes:

[0163] Determine the point of collision between the light ray and the reflecting surface;

[0164] When the collision point is close to the edge of the triangle corresponding to the reflecting surface, it is determined whether the edge is the saved diffraction edge; the collision point is determined to be close to the edge of the triangle corresponding to the reflecting surface based on the transformed centroid coordinates of the collision point.

[0165] If the edge is one of a plurality of diffraction edges that are preserved, then two shadow regions are determined according to the two reflecting surfaces corresponding to the diffraction edge. When the light is in either of the shadow regions, diffraction is determined to occur.

[0166] To enable light to track possible diffraction paths, a diffraction check is added at each collision. Figure 9 A schematic diagram of random diffraction provided for an embodiment of the present invention, such as... Figure 9 As shown, the process first locates the intersection point of the incident ray and the reflecting surface, i.e., the collision point, and determines whether this collision point is close to the edge of the triangle (i.e., the reflecting surface). When the collision point is close to the edge of the triangle, it is further determined whether this edge is a diffraction edge, which can be determined by reading pre-stored diffraction edge information. When the collision point is close to a diffraction edge, a diffraction point is created at the position closest to the intersection point on that diffraction edge, and a new diffracted ray is emitted from that diffraction point, thus continuing to track that diffracted ray.

[0167] When determining whether a collision point is close to the edge of a triangle, the coordinates of the collision point can be converted to the centroid coordinates of the triangle. The proximity of the collision point to the triangle's edge is then determined by comparing the centroid coordinates with preset values. Optionally, a coordinate system can be established with one side of the triangle as the u-axis and another side as the v-axis. The centroid coordinates can be represented as (u, v). The representation of the centroid coordinates depends on the established triangle's coordinate system. When u is less than a first preset value (actually, u is close to 0), the collision point is close to the v-axis; when v is less than a second preset value (actually, v is close to 0), the collision point is close to the u-axis; when the sum of u and v is close to 1, the collision point is close to the third side.

[0168] A second condition can be used to determine whether diffraction will occur. Figure 10 This is a top view schematic diagram of random diffraction provided in an embodiment of the present invention, as shown below. Figure 10 As shown, an example of diffraction ray generation is illustrated from another perspective. For each diffraction edge, extending the two triangles that make up it outward from the diffraction edge can form, as shown... Figure 10The diagram shows two shaded areas. To simplify calculations, we can consider only the diffraction between the two shaded areas. For example, light can only diffract from shaded area 1 to shaded area 2, or vice versa. If the incident light is in other areas, no new diffracted light is generated. Generating diffracted light based on this rule allows us to allocate limited computational resources as much as possible to diffraction paths that have a significant impact on the listener's perception, i.e., paths from one shaded area to another. For example, when the incident light is in the blank area in the lower right corner, reflection will occur, and the reflected light can propagate to shaded area 1. Even without considering diffraction, this will not have a significant impact on the listener's perception.

[0169] By determining whether diffraction occurs as described above, the accuracy of determining whether diffraction will occur during a collision can be improved, thereby making the determined diffraction path more accurate.

[0170] Figure 11 A schematic diagram of a path report provided in an embodiment of the present invention, such as... Figure 11 As shown, the reported path includes the reflection path and the diffraction path. In the listener tracking module 121, each time a light ray collides and diffracts, the current path is reported to the reflection tree construction module 122. The reported path is in the form of {start node, end node}, indicating that these two nodes can affect the information received by the listener and that they are interconnected. For example, when a light ray originates from the listener's position and collides with reflecting surface A, the reported path is {listener, reflecting surface A}. If the light ray also diffracts at the diffraction edge B, an additional path {listener, diffraction edge B} is reported. If a light ray emitted from reflecting surface A continues to propagate, collides with reflecting surface C, and diffracts at the diffraction edge D, the reported paths are {reflecting surface A, reflecting surface C}, {reflecting surface A, diffraction edge D}. The light emitted from the diffraction edge B continues to propagate and collides with the reflecting surface E, and diffracts at the diffraction edge F. The reported path is {diffraction edge B, reflecting surface E}, {diffraction edge B, diffraction edge F}.

[0171] The ray diffraction in the listener tracking module only provides basic path possibilities. That is, each ray can be transmitted based on each reported path, but the specific combination path of diffraction and reflection paths from the sound source location to the listener location (that is, the sound wave diffraction behavior) needs to be determined based on the reflection tree.

[0172] The reflection tree construction module 122 abstracts all reflecting surfaces and diffraction edges into nodes, and several nodes connected together can form a sound wave transmission path. When the scene model is large, processing and storing all nodes is impractical because the total number of paths they form increases exponentially. It is also unnecessary, as only a small portion of the nodes may affect the current listener's location. This is why this application uses the calculation results of the listener tracking module 121 to track which nodes the light rays originating from the listener pass through, and constructs the reflection tree only for these nodes.

[0173] like Figure 5 As shown, the output of the listener tracking module 121 is also input to the sound source tracking module 123, so that the sound source tracking module 123 can determine multiple first propagation paths from the sound source position to the listener position based on the collision information and the sound source position. Optionally, the sound source tracking module 123 can generate the propagation path and echo map from the sound source to the listener based on the sound source position, scene model, and reflectivity information. The tracked light rays originate from the listener position, so what is actually calculated is the propagation path from the listener position to the sound source position, and it is equivalent to assuming that there is a common propagation path from the sound source position to the listener position.

[0174] Specifically, a set of light rays is randomly emitted from the listener's position, each carrying the same initial energy E0. However, the sound source tracking module 123 no longer needs to calculate the intersection between the light rays and the scene model, because the output of the listener tracking module 121 already contains this collision information. For each collision point, it is only necessary to calculate the propagation path between it and the sound source position.

[0175] Figure 12 This is a schematic diagram illustrating a specific process of a sound source tracking module provided in an embodiment of the present invention. Figure 12 The execution steps determine the echo map corresponding to the first propagation path. For the light ray at the collision point, it can be determined whether diffraction occurs at the collision point. If diffraction occurs, the light energy is updated, i.e., the energy of the diffracted light ray. If no diffraction occurs, it can also be determined whether there is an obstruction between the collision point and the sound source location. If there is no obstruction, the path between the collision point and the sound source location can be determined. If there is an obstruction, the next collision point is processed. Once the path between the collision point and the sound source location is determined, the propagation path between the corresponding sound source location and the listener location can be obtained, and thus multiple paths between the sound source location and the listener location can be obtained. For the propagation path between the sound source location and the listener location, the echo map can be calculated, including: the energy received by the sound source, the arrival time and direction of the light ray reaching the sound source. When calculating the echo map, it can be determined based on the reflectivity and scattering rate of the reflecting surface where the collision point is located.

[0176] Figure 13This is a schematic diagram of a specular reflection path provided in an embodiment of the present invention; Figure 14 This is a schematic diagram of a scattering path provided in an embodiment of the present invention. Specifically, firstly, based on the reflectivity α of the reflecting surface... s The remaining energy α after reflection is obtained. s E, where E is the energy currently carried by the ray. α s E is further divided into two parts: mirror energy and scattering energy, where the scattering energy is α. scattering α s E, α scattering This is the scattering coefficient of the current collision surface, and the remaining part (1-α) scattering )α s E is the mirror energy. When there are no other obstructions between the collision point and the sound source, there exists a propagation path between them, and its direction originates from the location corresponding to the collision point. The mirror reflection angle of this ray is determined to see if the mirrored ray can pass through the sound source. If so, the path is considered a mirror reflection path, and the mirror energy of this ray is (1-α). scattering )α s If all energy E is emitted to the sound source, then the path is considered a scattering path, and a portion of the scattered energy of the light ray is emitted to the sound source. Assuming the scattering proportion is α, then the energy emitted to the sound source is αα. scattering α s The value of E. α can be given by the following formula:

[0177]

[0178] Where θ is the angle between the scattered ray (the ray pointing from the collision point to the sound source) and the normal vector of the reflecting surface, d is the distance from the collision point to the center of the sound source, and r is the radius of the sound source.

[0179] The cumulative propagation distance of the current light ray is added to the distance between the collision point and the sound source to obtain the total distance the light ray travels from the listener to the sound source. Based on this distance and the exponential attenuation model, the air absorption coefficient α of the current path is calculated. air This energy is then multiplied by the energy emitted from the collision point to the sound source to obtain the energy α received by the sound source. air (1-α scattering )α s E (Mirror Path) or α air αα scattering α s E (scattering path).

[0180] The energy received by the sound source, the total time it takes for the light ray to travel from the listener to the sound source, and the direction in which the light ray is emitted from the listener are saved to the echo map (since it is a reverse ray tracing, the direction of arrival is the opposite of the direction in which the light ray is emitted from the listener).

[0181] When light rays collide, if diffraction occurs simultaneously, it is necessary to continue tracking the reflected and diffracted rays. After calculating the energy received by the sound source, the energy received by the sound source, the energy of the diffracted rays, and the energy absorbed by the reflecting surface are subtracted from the energy carried by the light rays to obtain the updated light energy, which is the energy of the reflected rays after the collision. This updated light energy is used to determine if it reaches a lower limit. If it does not reach the lower limit, this energy is used for calculation at the next collision point, thus continuing to track the reflected rays. On the other hand, if diffraction occurs, the energy of the diffracted rays can also be obtained as the updated light energy, which is the energy of the diffracted rays after the collision. This updated light energy is also used to determine if it reaches a lower limit. If it does not reach the lower limit, this energy is used for calculation at the next collision point, thus continuing to track the diffracted rays. Optionally, the energy of the diffracted rays can be the product of the energy carried by the light rays and a fixed coefficient, which is not specifically limited here.

[0182] Determine if the updated ray energy has reached the lower limit and decide whether to continue calculating subsequent collision points for that ray. There's no need to calculate the intersection of the new ray with the scene model here, as the reflection direction and the next collision point have already been calculated in the sound source tracking module. You can directly begin calculating the propagation path between the next collision point and the sound source.

[0183] Through the sound source tracing process described above, the first propagation path from the sound source to the listener can be obtained. This first propagation path is mostly a scattering path, and the probability of it being a specular reflection path is relatively small. Furthermore, the above process does not determine the diffraction path. Therefore, a combined path of specular reflection and diffraction can be modeled based on a reflection tree. This part will be explained in detail below.

[0184] Optionally, a reflection tree is constructed based on the reported path, including:

[0185] The reported path is deduplicated to obtain a deduplicated path; the reported path includes a start node and an end node; the start node or the end node is a reflecting surface or a diffraction edge;

[0186] The listener is identified as the root node of the reflection tree, and the following steps are repeated until the maximum depth of the reflection tree reaches a preset depth:

[0187] For each node in the reflection tree, find the target node that the node can connect to from all the deduplicated paths, and determine the target node as the child node of the node.

[0188] When constructing a reflection tree based on the reported paths, all unique paths can be counted from the reported paths, which is to perform deduplication and construct the reflection tree. Figure 11 The reporting path is shown in its format, including the start node and the end node.

[0189] Figure 15 This is a schematic diagram of a reflection tree provided in an embodiment of the present invention. The reflection tree is a tree structure, with the root node representing the listener. Each node has several child nodes, representing a node in the sound propagation path. This node can be a reflecting surface or a diffraction edge. For each leaf node in the reflection tree, tracing back the path to the root node corresponds to a possible sound propagation path in the scene model. Specifically, when constructing the reflection tree, the listener is first determined as the root node. Then, from the deduplicated paths, a target node that the root node can connect to is found, and this target node is determined as a child node of the root node. The above steps are repeated for the child nodes of the root node to determine the reflection tree. That is, when constructing the reflection tree, each node is found to connect to from all deduplicated paths, and a corresponding child node is added to each node. This step is repeated for each child node until the maximum depth of the reflection tree reaches a preset depth. The value of the preset depth is not specifically limited here.

[0190] By determining the reflection tree based on the reporting nodes, each sound propagation path can be intuitively identified, which facilitates the subsequent determination of a combined path containing reflection and diffraction paths based on the reflection tree.

[0191] Optionally, the method further includes:

[0192] When determining the reflection tree, if the node is the root node, the spatial position corresponding to the root node is determined as the listener's position;

[0193] When the node is a reflective surface, determine the mirror position of the spatial position of the parent node of the node relative to the reflective surface, and determine the mirror position as the spatial position of the node.

[0194] When the node is a diffraction edge, the center position of the diffraction edge is determined as the diffraction point or the spatial position corresponding to the node.

[0195] While constructing the reflection tree, the spatial position of each node can be calculated, thus determining the reflection point position based on the spatial position of each node. The spatial position of the root node is the listener's position. For other nodes, when a node is a reflecting surface, its spatial position is the mirror image of its parent node relative to that reflecting surface. For nodes corresponding to diffraction edges, the center of the diffraction edge is determined as the spatial position of that node.

[0196] Figure 16 This is a schematic diagram of a mirror position and propagation path provided in an embodiment of the present invention, as shown below. Figure 16 As shown, sound propagates from the N+2 level node to the N+1 level node. The N+1 level node is the diffraction edge. The diffracted light propagates to the N level node, which is the reflecting surface. After reflection, the light propagates to the N-1 level node, which is the parent node of the N level node. For the N level node, since it is the reflecting surface, its spatial position is the mirror image of its parent node relative to the reflecting surface. This mirror image is the spatial position of the N level node.

[0197] In this way, when performing path modeling, the actual path of sound propagation from the child node to the parent node after reflection by the current reflecting surface can be equivalent to the propagation path from the child node position to the current reflecting node position (the parent node's mirror image). For nodes on other diffraction edges, the center position of the diffraction edge is directly used as the spatial position of the current node.

[0198] By determining the spatial location of each node, it becomes easier to determine the accurate propagation path later, thus eliminating the need for further calculations during the path search phase.

[0199] Optionally, the reflection path in the second propagation path is a specular reflection path; multiple second propagation paths from the sound source location to the listener location are determined based on the reflection tree, including:

[0200] For each sound source, determine the reflecting surface and diffraction edge where the light emitted from the sound source location collides with the scene model for the first time, so as to obtain a list of visible nodes of the sound source.

[0201] For each leaf node in the reflection tree, when the leaf node is in the sound source visible list, it is determined whether the propagation path corresponding to the leaf node is a valid path;

[0202] When the leaf node is not in the visible list of sound sources, the propagation path corresponding to the leaf node is determined to be an invalid path;

[0203] The second propagation path is determined based on the established legal path; the second propagation path is a path obtained by sequentially connecting the sound source location, the reflection points or diffraction points corresponding to each node in the legal path, and the listener's location.

[0204] After the reflection tree is constructed, for each sound source, the path search module 124 searches for all legal paths from it. Figure 17 This is a schematic diagram illustrating a specific process of a path search module provided in an embodiment of the present invention, such as... Figure 17As shown, firstly, random light rays are emitted from the sound source. The reflective surfaces and diffraction edges where the light rays first collide with the scene model are counted, resulting in a unique list of visible sound source nodes. Nodes in this list represent visible reflective surfaces or diffraction edges of the sound source. Then, for each leaf node in the reflection tree, it is first determined whether the leaf node is in the list of visible sound source nodes. If it is not in the list, the sound propagation path corresponding to that leaf node is an invalid path. If it is in the list, the path corresponding to that leaf node is further determined to be a valid path.

[0205] By determining a list of visible sound source nodes, and then identifying the leaf nodes within that list, we can further determine whether the sound propagation path corresponding to that leaf node is a valid path. This approach can improve the accuracy of the determined second propagation path while reducing computational complexity.

[0206] Optionally, determining whether the propagation path corresponding to the leaf node is a valid path includes:

[0207] If every child node from the leaf node to the root node satisfies the target condition, then the path from the leaf node to the root node is determined to be a valid path; otherwise, it is an invalid path.

[0208] When the node is a reflective surface, the target condition is: the reflection point corresponding to the reflective surface is within the triangle corresponding to the reflective surface and the corresponding path is not obstructed; the position of the reflection point is related to the spatial position corresponding to the reflective surface and the spatial position of the next level node.

[0209] When the node is a diffraction edge, the target conditions are: the corresponding path is not occluded, and the preceding and following nodes of the node are located in two different shadow areas corresponding to the node.

[0210] When determining whether the propagation path corresponding to a leaf node is a valid path, the leaf node can be designated as the current node. If the current node is a reflection node (reflecting surface), then the location of the reflection point can be determined, such as... Figure 16As shown, the reflection point is the intersection of the line connecting the spatial position of the next-level node and the spatial position of the current node with the current reflecting surface. This reflection point is the intersection of the sound propagation path and the reflecting surface. After determining the reflection point, it is then judged that it must be within the triangle corresponding to the reflecting surface to be a valid path. Furthermore, the path formed by the spatial position of the next-level node and the reflection point can be checked for occlusion by other objects within the scene model. If it is not occluded, the above steps are repeated for the parent node of the current node until all child nodes from the leaf node to the root node satisfy the condition that the reflection point is within the triangle corresponding to the reflecting surface and the corresponding path is not occluded by other objects. Then, the propagation path corresponding to the leaf node is determined to be a valid path. In other words, connecting the reflection points and diffraction points corresponding to all propagation nodes in the propagation path yields a complete propagation path. Only when this propagation path is not occluded can it be determined to be a valid path.

[0211] When a node is a diffraction edge, it is necessary not only to determine that the corresponding path is not occluded, but also to determine that the node's predecessor and successor nodes are located in two different shadow regions corresponding to the current node. The two different shadow regions corresponding to the current node refer to the two different shadow regions corresponding to the diffraction edge. By restricting this condition, we can focus only on diffraction paths that have a significant impact on the auditory experience, and not on diffraction paths that have a smaller impact on the auditory experience, thereby reducing the amount of computation.

[0212] By determining whether the reflection point is located within the triangle corresponding to the reflecting surface, and whether there is any obstruction along the corresponding path, the accuracy of determining whether the propagation path is a legitimate path can be improved.

[0213] Optionally, determining the echo map corresponding to the second propagation path includes:

[0214] For the sound source, determine the initial energy corresponding to the sound source;

[0215] For each second propagation path, the distance attenuation coefficient and air absorption coefficient are determined based on the total length of the second propagation path. The reflection attenuation coefficient of each reflecting surface in the second propagation path is determined. The reflection attenuation coefficients of each reflecting surface are multiplied to obtain the overall reflection attenuation coefficient. The diffraction coefficient is determined based on the diffraction edge information of each diffraction edge in the second propagation path. The reflection attenuation coefficient of the reflecting surface is related to the reflectivity and scattering rate of the reflecting surface.

[0216] The multiplication result of the initial energy, the distance attenuation coefficient, the air absorption coefficient, the overall reflection attenuation coefficient, and the diffraction coefficient is determined as the energy received by the listener;

[0217] The direction in which the last node points to the listener's position is defined as the receiving direction;

[0218] The energy received by the listener, the receiving direction, and the propagation time corresponding to the second propagation path are determined as the echo map corresponding to the second propagation path.

[0219] The path modeling module 125 models all the second propagation paths searched by the path search module 124 and obtains the echo map corresponding to each second propagation path. This result is merged with the echo map obtained by the sound source tracing module 123 and used for subsequent filter synthesis. For each propagation path, the listener's received energy, arrival time, and direction of arrival are saved to the echo map.

[0220] Figure 18 This is a schematic diagram of a diffraction path modeling provided in an embodiment of the present invention, as shown below. Figure 18 As shown, the left half is the second propagation path found in the reflection tree, and the right half is the actual sound propagation path corresponding to this second propagation path. The path modeling module 125 is used to calculate the arrival time, direction of arrival, and listener received energy for each second propagation path.

[0221] Modeling the energy received by the listener mainly consists of several parts: distance attenuation, air absorption, reflection attenuation, and diffraction modeling. Among these, the distance attenuation coefficient α... d The total length d of the propagation path can be obtained using an inverse distance model. The air absorption coefficient α air It can be calculated using a distance and exponential attenuation model. The reflection attenuation coefficient of each reflecting surface can be calculated from the reflectivity α of that reflecting surface. s and scattering rate α scattering We obtain it through the following formula: (1-α) scattering )α s The overall reflection attenuation coefficient α is obtained by multiplying the reflection attenuation coefficients of all reflecting surfaces along the second propagation path. reflection Modeling diffraction can be done using a unified theory of diffraction (UTD) or a biot-tolstoy model (BTM), and this invention does not limit this approach. The UTD or BTM method requires diffraction edge information (stored in diffraction edge data 112), such as triangle normal vectors, diffraction surface vectors, diffraction edge lengths, and diffraction angles. Based on the diffraction edge information corresponding to each diffraction edge in the second propagation path, the diffraction coefficient α corresponding to that second propagation path can be determined. diffraction This represents the ratio of diffraction energy to incident energy. Furthermore, the direction from the last node pointing to the listener's position can be defined as the receiving direction, or the direction of arrival.

[0222] The energy ultimately received by the listener can be represented as α. d α air α reflection α diffraction E1, where E1 represents the initial energy corresponding to the sound source. The ratio of the initial energy E1 corresponding to the sound source to E0 in the sound source tracing module 123 reflects the proportion of the two modeling methods, reflection tree and random ray tracing, in the final result. This ratio can be adjusted according to the actual situation. In a specific embodiment, the calculation of the above attenuation coefficient can also be divided into different frequency bands to obtain more accurate modeling results.

[0223] Figure 19 This is a schematic diagram of an echo map provided in an embodiment of the present invention, such as... Figure 19 As shown, the echoogram generated by the sound source tracing module 123 and the path modeling module 125 records all reflection, scattering, or diffraction paths from the sound source to the listener. Each point in the echoogram represents the energy E carried by a possible propagation path i when it reaches the listener. i Propagation time t i And the horizontal and vertical angles θ of the direction of arrival. i , Because the reflectivity of the reflecting surface may vary for signals of different frequencies, this energy value can also be divided into multiple frequency bands.

[0224] By taking into account factors such as distance attenuation coefficient, air absorption coefficient, overall reflection attenuation coefficient, and diffraction coefficient, the accuracy of the determined listener's received energy is relatively high.

[0225] Figure 20 This is a schematic diagram of the structure of a filter synthesis module provided in an embodiment of the present invention, as shown below. Figure 20 As shown, the filter synthesis module 126 synthesizes the echo map into an Ambisonic impulse response, which is divided into several steps: Ambisonics encoding 210, filter generation 211, and subband synthesis 212.

[0226] The Ambisonics encoding module 210 first performs Ambisonic encoding on the echo map. Based on the arrival direction of each propagation path, the corresponding Ambisonics coefficients can be obtained. Multiplying this by the corresponding energy yields the Ambisonics strength:

[0227]

[0228] in, These are spherical harmonic basis functions:

[0229]

[0230] Among them, P l |m| It is a combined Legendre function.

[0231] Figure 21 An example diagram for reconstructing the frequency band impulse response from an echo map, as provided in this embodiment of the invention, is shown below. Figure 21 As shown, the filter generation module 211 processes each Ambisonics channel in the Ambisonics echo map separately. For each channel, based on the propagation time and intensity of each path in different frequency bands, an impulse response for the corresponding frequency band is generated. Specifically, the propagation time and corresponding Ambisonics intensity of each point in the echo map are first read. Then, the sampling point position corresponding to the propagation time is found in the impulse response of the corresponding Ambisonics channel and frequency band. Subsequently, a pulse with a peak value equal to the corresponding intensity is inserted at this sampling point. Depending on actual needs, when the sampling point position corresponding to the propagation time is not an integer, interpolation can also be performed on the corresponding intensity within a certain time range before and after that position, such as using Lagrange interpolation. This invention does not impose any restrictions on this.

[0232] The subband synthesis module 212 can reconstruct the full-band impulse response from the subband impulse response using an FIR filter bank or a Linkwitz-Riley filter bank based on a Biquad IIR filter, and the present invention does not limit this.

[0233] Figure 22 This is a schematic diagram of an acoustic ray tracing device provided in an embodiment of the present invention. The device 220 includes:

[0234] The listener tracking module 2201 is used to determine the collision information and reporting path corresponding to the light emitted from the listener's position; the collision information is the information of each collision point obtained by the light colliding with the scene model multiple times;

[0235] The reflection tree construction module 2202 is used to construct a reflection tree based on the reported path; the reported path is a path composed of the reflecting surface or diffraction edge corresponding to the collision point.

[0236] The sound source tracking module 2203 is used to determine, for each sound source, multiple first propagation paths from the sound source location to the listener location based on the collision information and the sound source location;

[0237] The path search module 2204 is used to determine multiple second propagation paths from the sound source location to the listener location based on the reflection tree; the second propagation path is a combination path of specular reflection path and diffraction path;

[0238] The sound source tracking module 2203 is also used to determine the echo map corresponding to the first propagation path for each sound source;

[0239] The path modeling module 2205 is used to determine the echo map corresponding to the second propagation path;

[0240] The filter synthesis module 2206 is used to determine the filter corresponding to the sound source based on the echo map;

[0241] Processing module 2207 is used to process the unspatialized audio stream of each sound source according to the filter corresponding to each sound source to determine the reverberation signal.

[0242] Optionally, when the listener tracking module 2201 determines the collision information and reporting path corresponding to the light emitted from the listener's location, and when the reflection tree construction module 2202 constructs a reflection tree based on the reported path, it is specifically used for:

[0243] When at least one of the following conditions is met, the listener tracking module 2201 performs ray tracing on the light rays emitted from the listener's location to determine collision information and a reporting path, and the reflection tree construction module 2202 constructs a reflection tree based on the reported path:

[0244] The listener's position changes, the scene model changes, and the reflectivity of any reflective surface in the scene model changes.

[0245] Optionally, the scene model is represented using triangulation; the device further includes: a preprocessing module, used for:

[0246] Determine the diffraction edge in the scene model; the diffraction edge is the coincident side of two non-coplanar triangles;

[0247] Determine and save diffraction edge information; the diffraction edge information corresponds to the diffraction edge; wherein, when the dynamic geometry in the scene model changes, the corresponding diffraction edge information is updated;

[0248] Accordingly, when the listener tracking module 2201 determines the collision information and reporting path corresponding to the light emitted from the listener's location, it is specifically used for:

[0249] Ray tracing is performed on the light emitted from the listener's position to determine collision information, and the reporting path is determined based on the diffraction edge information.

[0250] Optionally, the reporting path includes a reflection path and a diffraction path; when the listener tracking module 2201 performs ray tracing on the light emitted from the listener's position to determine collision information and determines the reporting path based on the diffraction edge information, it is specifically used for:

[0251] For each ray of light, repeat the following steps to determine the collision information each time the ray collides, until the condition for ending the tracking of the ray is met:

[0252] When the light ray collides with the scene model, the collision information for this collision is determined;

[0253] When it is determined that the tracking of the ray will not end, the reflection path is determined and the direction of the reflected ray is calculated to determine whether a collision will occur with another reflective surface in the scene model based on the direction of the reflected ray.

[0254] Furthermore, when it is determined that the tracking of the light ray will not end, it is determined whether diffraction will occur based on the diffraction edge information. When diffraction occurs, the diffraction path is determined, and the direction of the diffracted light ray is calculated to determine whether it will collide with another reflective surface in the scene model based on the direction of the diffracted light ray.

[0255] Optionally, when determining whether diffraction will occur based on the diffraction edge information, the listener tracking module 2201 is specifically used for:

[0256] Determine the point of collision between the light ray and the reflecting surface;

[0257] When the collision point is close to the edge of the triangle corresponding to the reflecting surface, it is determined whether the edge is the saved diffraction edge; the collision point is determined to be close to the edge of the triangle corresponding to the reflecting surface based on the transformed centroid coordinates of the collision point.

[0258] If the edge is one of a plurality of diffraction edges that are preserved, then two shadow regions are determined according to the two reflecting surfaces corresponding to the diffraction edge. When the light is in either of the shadow regions, diffraction is determined to occur.

[0259] Optionally, when constructing a reflection tree based on the reported path, the reflection tree construction module 2202 is specifically used for:

[0260] The reported path is deduplicated to obtain a deduplicated path; the reported path includes a start node and an end node; the start node or the end node is a reflecting surface or a diffraction edge;

[0261] The listener is identified as the root node of the reflection tree, and the following steps are repeated until the maximum depth of the reflection tree reaches a preset depth:

[0262] For each node in the reflection tree, find the target node that the node can connect to from all the deduplicated paths, and determine the target node as the child node of the node.

[0263] Optionally, the device further includes: a spatial location determination module, used for:

[0264] When determining the reflection tree, if the node is the root node, the spatial position corresponding to the root node is determined as the listener's position;

[0265] When the node is a reflective surface, determine the mirror position of the spatial position of the parent node of the node relative to the reflective surface, and determine the mirror position as the spatial position of the node.

[0266] When the node is a diffraction edge, the center position of the diffraction edge is determined as the diffraction point or the spatial position corresponding to the node.

[0267] Optionally, the reflection path in the second propagation path is a specular reflection path; when determining multiple second propagation paths from the sound source location to the listener location based on the reflection tree, the path search module 2204 is specifically used for:

[0268] For each sound source, determine the reflecting surface and diffraction edge where the light emitted from the sound source location collides with the scene model for the first time, so as to obtain a list of visible nodes of the sound source.

[0269] For each leaf node in the reflection tree, when the leaf node is in the sound source visible list, it is determined whether the propagation path corresponding to the leaf node is a valid path;

[0270] When the leaf node is not in the visible list of sound sources, the propagation path corresponding to the leaf node is determined to be an invalid path;

[0271] The second propagation path is determined based on the established legal path; the second propagation path is a path obtained by sequentially connecting the sound source location, the reflection points or diffraction points corresponding to each node in the legal path, and the listener's location.

[0272] Optionally, when determining whether the propagation path corresponding to the leaf node is a valid path, the path search module 2204 is specifically used for:

[0273] If every child node from the leaf node to the root node satisfies the target condition, then the path from the leaf node to the root node is determined to be a valid path; otherwise, it is an invalid path.

[0274] When the node is a reflective surface, the target condition is: the reflection point corresponding to the reflective surface is within the triangle corresponding to the reflective surface and the corresponding path is not obstructed; the position of the reflection point is related to the spatial position corresponding to the reflective surface and the spatial position of the next level node.

[0275] When the node is a diffraction edge, the target conditions are: the corresponding path is not occluded, and the preceding and following nodes of the node are located in two different shadow areas corresponding to the node.

[0276] Optionally, when determining the echo map corresponding to the second propagation path, the path modeling module 2205 is specifically used for:

[0277] For the sound source, determine the initial energy corresponding to the sound source;

[0278] For each second propagation path, the distance attenuation coefficient and air absorption coefficient are determined based on the total length of the second propagation path. The reflection attenuation coefficient of each reflecting surface in the second propagation path is determined. The reflection attenuation coefficients of each reflecting surface are multiplied to obtain the overall reflection attenuation coefficient. The diffraction coefficient is determined based on the diffraction edge information of each diffraction edge in the second propagation path. The reflection attenuation coefficient of the reflecting surface is related to the reflectivity and scattering rate of the reflecting surface.

[0279] The multiplication result of the initial energy, the distance attenuation coefficient, the air absorption coefficient, the overall reflection attenuation coefficient, and the diffraction coefficient is determined as the energy received by the listener;

[0280] The direction in which the last node points to the listener's position is defined as the receiving direction;

[0281] The energy received by the listener, the receiving direction, and the propagation time corresponding to the second propagation path are determined as the echo map corresponding to the second propagation path.

[0282] The acoustic ray tracing device 220 provided in this embodiment of the invention can achieve the above-mentioned... Figure 3 The acoustic ray tracing method shown in the embodiment has a similar implementation principle and technical effect, and will not be described again here.

[0283] Figure 23 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. Figure 23 As shown, the electronic device provided in this embodiment includes at least one processor 2301 and a memory 2302. The processor 2301 and the memory 2302 are connected via a bus 2303.

[0284] In a specific implementation, at least one processor 2301 executes computer execution instructions stored in memory 2302, causing at least one processor 2301 to execute the method in the above method embodiment.

[0285] The specific implementation process of processor 2301 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0286] Optionally, the electronic device can be a head-mounted display device.

[0287] In the above Figure 23 In the illustrated embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0288] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage.

[0289] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0290] This invention also provides a wearable device, including a processing unit; the processing unit is used to implement the method described in the above method embodiments.

[0291] This invention also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the method described in the above embodiments.

[0292] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the method described in the above method embodiments.

[0293] The aforementioned computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0294] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0295] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0296] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0297] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0298] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. An acoustic ray tracing method, characterized in that, include: Determine the collision information and reporting path corresponding to the light emitted from the listener's location, and construct a reflection tree based on the reporting path; the collision information is the information of each collision point obtained by the light colliding with the scene model multiple times; The reporting path is the path formed by the reflecting surface or diffraction edge corresponding to the collision point; For each sound source, multiple first propagation paths from the sound source location to the listener location are determined based on the collision information and the sound source location, and multiple second propagation paths from the sound source location to the listener location are determined based on the reflection tree; The second propagation path is a combination of the reflection path and the diffraction path; For each sound source, determine the echo map corresponding to the first propagation path, and determine the echo map corresponding to the second propagation path; The filter corresponding to the sound source is determined based on the echo map, and the unspatialized audio stream of the corresponding sound source is processed according to the filter corresponding to each sound source to determine the reverberation signal.

2. The method according to claim 1, characterized in that, Determine the collision information and reporting path corresponding to the light emitted from the listener's location, and construct a reflection tree based on the reporting path, including: When at least one of the following conditions is met, the collision information and reporting path corresponding to the light emitted from the listener's location are determined, and a reflection tree is constructed based on the reporting path: The listener's position changes, the scene model changes, and the reflectivity of any reflective surface in the scene model changes.

3. The method according to claim 1, characterized in that, The scene model is represented using triangulation; the method further includes: Determine the diffraction edge in the scene model; the diffraction edge is the coincident side of two non-coplanar triangles; Determine and save diffraction edge information; the diffraction edge information corresponds to the diffraction edge; wherein, when the dynamic geometry in the scene model changes, the corresponding diffraction edge information is updated; Accordingly, the collision information and reporting path corresponding to the light emitted from the listener's location are determined, including: Ray tracing is performed on the light emitted from the listener's location to determine collision information, and the reporting path is determined based on the diffraction edge information.

4. The method according to claim 3, characterized in that, The reporting path includes the reflection path and the diffraction path; Ray tracing is performed on the light emitted from the listener's location to determine collision information, and the reporting path is determined based on the diffraction edge information, including: For each ray of light, repeat the following steps to determine the collision information each time the ray collides, until the condition for ending the tracking of the ray is met: When the light ray collides with the scene model, the collision information for this collision is determined; When it is determined that the tracking of the ray will not end, the reflection path is determined and the direction of the reflected ray is calculated to determine whether a collision will occur with another reflective surface in the scene model based on the direction of the reflected ray. Furthermore, when it is determined that the tracking of the light ray will not end, it is determined whether diffraction will occur based on the diffraction edge information. When diffraction occurs, the diffraction path is determined, and the direction of the diffracted light ray is calculated to determine whether it will collide with another reflective surface in the scene model based on the direction of the diffracted light ray.

5. The method according to claim 3, characterized in that, Determining whether diffraction will occur based on the diffraction edge information includes: Determine the point of collision between the light ray and the reflecting surface; When the collision point is close to the edge of the triangle corresponding to the reflecting surface, it is determined whether the edge is the saved diffraction edge; the collision point is determined to be close to the edge of the triangle corresponding to the reflecting surface based on the transformed centroid coordinates of the collision point. If the edge is one of a plurality of diffraction edges that are preserved, then two shadow regions are determined according to the two reflecting surfaces corresponding to the diffraction edge, and when the light is in either of the shadow regions, diffraction is determined to occur.

6. The method according to claim 3, characterized in that, Constructing a reflection tree based on the reported path includes: The reported path is deduplicated to obtain a deduplicated path; the reported path includes a start node and an end node; the start node or the end node is a reflecting surface or a diffraction edge; The listener is identified as the root node of the reflection tree, and the following steps are repeated until the maximum depth of the reflection tree reaches a preset depth: For each node in the reflection tree, find the target node that the node can connect to from all the deduplicated paths, and determine the target node as the child node of the node.

7. The method according to claim 6, characterized in that, The method further includes: When determining the reflection tree, if the node is the root node, the spatial position corresponding to the root node is determined as the listener's position; When the node is a reflective surface, determine the mirror position of the spatial position of the parent node of the node relative to the reflective surface, and determine the mirror position as the spatial position of the node. When the node is a diffraction edge, the center position of the diffraction edge is determined as the diffraction point or the spatial position corresponding to the node.

8. The method according to claim 7, characterized in that, The reflection path in the second propagation path is a specular reflection path; multiple second propagation paths from the sound source location to the listener location are determined based on the reflection tree, including: For each sound source, determine the reflecting surface and diffraction edge where the light emitted from the sound source location collides with the scene model for the first time, so as to obtain a list of visible nodes of the sound source. For each leaf node in the reflection tree, when the leaf node is in the sound source visible list, it is determined whether the propagation path corresponding to the leaf node is a valid path; When the leaf node is not in the visible list of sound sources, the propagation path corresponding to the leaf node is determined to be an invalid path; The second propagation path is determined based on the established legal path; the second propagation path is a path obtained by sequentially connecting the sound source location, the reflection points or diffraction points corresponding to each node in the legal path, and the listener's location.

9. The method according to claim 8, characterized in that, Determining whether the propagation path corresponding to the leaf node is a valid path includes: If every child node from the leaf node to the root node satisfies the target condition, then the path from the leaf node to the root node is determined to be a valid path; otherwise, it is an invalid path. When the node is a reflective surface, the target condition is: the reflection point corresponding to the reflective surface is within the triangle corresponding to the reflective surface and the corresponding path is not obstructed; the position of the reflection point is related to the spatial position corresponding to the reflective surface and the spatial position of the next level node. When the node is a diffraction edge, the target conditions are: the corresponding path is not occluded, and the preceding and following nodes of the node are located in two different shadow areas corresponding to the node.

10. The method according to claim 9, characterized in that, Determining the echo map corresponding to the second propagation path includes: For the sound source, determine the initial energy corresponding to the sound source; For each second propagation path, the distance attenuation coefficient and air absorption coefficient are determined based on the total length of the second propagation path. The reflection attenuation coefficient of each reflecting surface in the second propagation path is determined. The reflection attenuation coefficients of each reflecting surface are multiplied to obtain the overall reflection attenuation coefficient. The diffraction coefficient is determined based on the diffraction edge information of each diffraction edge in the second propagation path. The reflection attenuation coefficient of the reflecting surface is related to the reflectivity and scattering rate of the reflecting surface. The multiplication result of the initial energy, the distance attenuation coefficient, the air absorption coefficient, the overall reflection attenuation coefficient, and the diffraction coefficient is determined as the energy received by the listener; The direction in which the last node points to the listener's position is defined as the receiving direction; The energy received by the listener, the receiving direction, and the propagation time corresponding to the second propagation path are determined as the echo map corresponding to the second propagation path.

11. An acoustic ray tracing device, characterized in that, include: The listener tracking module is used to determine the collision information and reporting path of the light emitted from the listener's location; the collision information is the information of each collision point obtained by the light colliding with the scene model multiple times; A reflection tree construction module is used to construct a reflection tree based on the reported path; the reported path is a path composed of the reflecting surface or diffraction edge corresponding to the collision point. The sound source tracking module is used to determine, for each sound source, multiple first propagation paths from the sound source location to the listener location based on the collision information and the sound source location; The path search module is used to determine multiple second propagation paths from the sound source location to the listener location based on the reflection tree; the second propagation path is a combination path of specular reflection path and diffraction path; The sound source tracking module is also used to determine the echo map corresponding to the first propagation path for each sound source. The path modeling module is used to determine the echo map corresponding to the second propagation path; A filter synthesis module is used to determine the filter corresponding to the sound source based on the echo map; The processing module is used to process the unspatialized audio stream of each sound source according to the filter corresponding to each sound source to determine the reverberation signal.

12. An electronic device, characterized in that, include: At least one processor and memory; The memory stores computer-executed instructions; The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the method as described in any one of claims 1 to 10.

13. A wearable device, characterized in that, It includes a processing unit; the processing unit is used to perform the method as described in any one of claims 1 to 10.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1 to 10.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 10.

Images