Sound tracing-based real-time sound visualization device and method

Abstract

The present invention relates to a sound tracing-based real-time sound visualization device and method. The device comprises: a location tracking unit for tracking, in a three-dimensional space including at least a user and a sound source, location information of the user and the sound source in real time; a coordinate conversion unit for receiving the location information of the user and the sound source and converting same into coordinates in the three-dimensional space; a sound path determination unit for determining a candidate sound path reaching the user among a plurality of sound propagation paths generated from the sound source on the basis of spatial information of the three-dimensional space including physical attributes of materials constituting the three-dimensional space; and a sound visualization unit for visualizing and displaying at least a part of the candidate sound path in the three-dimensional space.

Description

Real-time sound visualization device and method based on sound tracing

[0001] The present invention relates to a real-time 3D sound visualization technology, and more particularly, to a real-time sound visualization technology based on sound tracing that provides a sound path in real time by visualizing it according to the positions of a sound source and a listener in a 3D space and the physical properties of materials constituting the 3D space.

[0002]

[0003] Recent advancements in mobile technology, graphics technology, and sensory input / output technologies have led to a rapid increase in interest in virtual reality (VR) technology. To support a realistic VR environment, not only visual spatial reproduction through virtual space but also high-quality auditory spatial reproduction is required. To achieve this spatial reproduction, multichannel audio systems or 3D sound technology utilizing the Head Related Transfer Function (HRTF) can be used.

[0004] However, multi-channel audio used for 3D sound reproduction has problems such as requiring a dedicated speaker system and the space required for the speaker system to be installed. Most 3D sound technologies based on the head-related transfer function use pre-calculated acoustic control filters or reproduce auditory spatial sensations within simple virtual spaces such as rectangular shoe boxes. However, these conventional 3D sound technologies have limitations in reproducing realistic sound because they do not reflect the effects of the surrounding spatial environment and the physical properties of object materials, including reflection, absorption, and propagation.

[0005] To overcome these limitations, 3D sound technologies based on geometric or numerical methods are being developed. Among the various geometric methods, a method that combines 3D graphics ray tracing technology and sound processing technology is called sound tracing. Sound tracing is a type of sound rendering technique that generates sound by tracing the sound propagation paths between the listener and the sound source in a 3D space.

[0006] The actual sound tracing process may require real-time processing synchronized with changing situations at each frame. Specifically, the various sound propagation paths generated and tracked during the sound tracing process are invisible to the user's eyes. However, in some cases, visualizing these paths can provide a more realistic 3D sound experience.

[0007]

[0008] [Prior Art Literature]

[0009] [Patent Document]

[0010] Korean Patent No. 10-1828908 (February 7, 2018)

[0011]

[0012] One embodiment of the present invention provides a real-time sound visualization technology based on sound tracing that visualizes and provides a sound path in real time according to the positions of a sound source and a listener in a three-dimensional space and the physical properties of a material forming the three-dimensional space.

[0013]

[0014] Among the embodiments, a real-time sound visualization device based on sound tracing includes a position tracking unit that tracks position information of a user and a sound source in real time in a three-dimensional space including at least a user and a sound source; a coordinate conversion unit that receives the position information of the user and the sound source and converts it into coordinates in the three-dimensional space; a sound path determination unit that determines a candidate sound path reaching the user among a plurality of sound propagation paths generated from the sound source based on spatial information of the three-dimensional space including physical properties of a material constituting the three-dimensional space; and a sound visualization unit that visualizes and displays at least a portion of the candidate sound paths in the three-dimensional space.

[0015] The above location tracking unit can track the location information of multiple users and sound sources in real time.

[0016] The above sound path determination unit can classify the plurality of sound propagation paths according to sound types including direct sound, diffraction sound, and reflected sound.

[0017] The above sound visualization unit can visualize and display the candidate sound path centered on a specific user or a specific sound source among the plurality of users and sound sources.

[0018] The above sound visualization unit can adjust the visualization conditions of the candidate sound path according to characteristic information of the specific user or specific sound source.

[0019] The above sound visualization unit can visualize the candidate sound path by distinguishing colors according to sound type.

[0020] The above sound visualization unit can visualize and display the candidate sound path in an augmented reality manner in an actual space corresponding to the three-dimensional space.

[0021]

[0022] The disclosed technology may have the following effects. However, this does not mean that a particular embodiment must include all or only the following effects, and thus the scope of the disclosed technology should not be construed as being limited thereby.

[0023] A real-time sound visualization device and method based on sound tracing according to one embodiment of the present invention tracks the positions of a user and a sound source in real time to update a sound path according to dynamic environmental changes, directly expresses the sound path in 3D space so that the user can intuitively understand the sound environment, and clearly expresses the characteristics of each sound path by visualizing direct sound, diffraction sound, reflected sound, etc. by distinguishing them by color.

[0024] In addition, in the case of the present invention, a sound path can be visualized centered on a specific user to provide a personal sound experience to the user, and at the same time, a sound path visualization condition can be adjusted according to the user's characteristic information (e.g., hearing level) to provide a customized experience to the user.

[0025]

[0026] Figure 1 is a diagram illustrating the pipeline of sound tracing.

[0027] Figure 2 is a drawing explaining the types of sound propagation paths.

[0028] Figure 3 is a drawing explaining mode switching of a conventional sound propagation unit.

[0029] Figure 4 is a drawing explaining a conventional sound propagation processing process.

[0030] FIG. 5 is a drawing illustrating a real-time sound visualization system according to the present invention.

[0031] Fig. 6 is a drawing explaining the system configuration of the acoustic visualization device of Fig. 5.

[0032] Figure 7 is a drawing explaining the functional configuration of the processor of Figure 6.

[0033] Figure 8 is a drawing illustrating one embodiment of an acoustic visualization unit according to the present invention.

[0034] Figure 9 is a flowchart illustrating a real-time sound visualization method based on sound tracing according to the present invention.

[0035]

[0036] The description of the present invention is merely an example for structural and functional explanation, and therefore, the scope of the present invention should not be construed as being limited by the embodiments described in the text. That is, since the embodiments can be modified in various ways and can take various forms, the scope of the present invention should be understood to include equivalents that can realize the technical idea. In addition, the purposes or effects presented in the present invention do not mean that a specific embodiment must include all of them or only such effects, and therefore, the scope of the present invention should not be construed as being limited thereby.

[0037] Meanwhile, the meaning of the terms described in this application should be understood as follows.

[0038] Terms such as "first" and "second" are intended to distinguish one component from another, and the scope of the rights should not be limited by these terms. For example, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component.

[0039] When a component is said to be "connected" to another component, it should be understood that while it may be directly connected to that other component, there may also be other components intervening. Conversely, when a component is said to be "directly connected" to another component, it should be understood that there are no other intervening components. Similarly, other expressions describing relationships between components, such as "between" and "directly between," or "adjacent to" and "directly adjacent to," should be interpreted similarly.

[0040] Singular expressions should be understood to include plural expressions unless the context clearly indicates otherwise, and terms such as "comprises" or "have" should be understood to specify the presence of a feature, number, step, operation, component, part or combination thereof, but not to exclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0041] For each step, the identifiers (e.g., a, b, c, etc.) are used for convenience of explanation and do not describe the order of the steps. The steps may occur in a different order than stated unless the context clearly dictates a specific order. That is, the steps may occur in the same order as stated, may be performed substantially simultaneously, or may be performed in the opposite order.

[0042] The present invention can be implemented as computer-readable code on a computer-readable recording medium. The computer-readable recording medium includes all types of recording devices that store data that can be read by a computer system. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices. Furthermore, the computer-readable recording medium can be distributed across network-connected computer systems, so that the computer-readable code can be stored and executed in a distributed manner.

[0043] Unless otherwise defined, all terms used herein have the same meaning as commonly understood by those of ordinary skill in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted to be consistent with their meaning within the context of the relevant technology, and should not be interpreted as having an idealized or overly formal meaning unless explicitly defined herein.

[0044]

[0045] Figure 1 is a diagram illustrating the pipeline of sound tracing.

[0046] Referring to Figure 1, the sound tracing pipeline can be comprised of sound propagation and sound generation (auralization) stages. Among the sound tracing processing stages, the sound propagation stage can be the most crucial step in providing immersion in virtual reality, and it can also be the stage with the highest computational complexity and the longest computational time. Furthermore, the acceleration of this stage can determine the real-time processing of sound tracing.

[0047] Specifically, the sound synthesis stage may be responsible for generating sound effects based on user interaction. For example, sound synthesis may process sounds generated when a user knocks on a door or drops an object, and may correspond to a technology commonly used in existing games, user interfaces, and the like.

[0048] The sound propagation stage may correspond to the stage of processing the acoustic signal according to the positions of the sound source and the listener in the 3D virtual space, and the materials that make up the 3D virtual space. For example, the sound propagation stage may perform acoustic simulation based on the impulse response (IR), which includes physical properties such as the size, shape, and material of a specific space. Here, the physical properties of the material may correspond to the degree of reflection, reflection-diffraction, refraction, and absorption of the sound according to the material, and may be classified according to materials such as concrete, grass, wood, and marble, for example. In other words, the sound propagation stage can calculate the distance, time, and resulting energy loss of the sound traveled according to the positions of the sound source and the listener in the 3D virtual space, and the physical properties of the materials that make up the 3D virtual space.

[0049] In one embodiment, the sound propagation step may correspond to a step that simulates the process of a synthesized sound being transmitted to a listener through a virtual reality, and may correspond to a step that processes the acoustic characteristics of the virtual reality (reflection coefficient, absorption coefficient, etc.) and the characteristics of sound (reflection, absorption, transmission, etc.) based on the geometric characteristics of the virtual reality (scene geometry).

[0050] Finally, the sound generation stage may be a stage that regenerates the input sound based on the configuration of the listener's speakers using the sound characteristic values ​​(reflection / transmission / absorption coefficients, distance attenuation characteristics, etc.) calculated in the sound propagation stage.

[0051]

[0052] Figure 2 is a drawing explaining the types of sound propagation paths.

[0053] Referring to FIG. 2, a direct path may correspond to a path that is transmitted directly between a listener and a sound source without any obstructions. A reflection path may correspond to a path in which sound is reflected after colliding with an obstacle and reaches the listener. A transmission path may correspond to a path in which sound transmits through an obstacle and is transmitted to the listener when there is an obstacle between the listener and the sound source.

[0054] Sound tracing can shoot rays from the respective locations of multiple sound sources. Each shot ray can then find the geometric objects it collides with and generate rays corresponding to reflection, transmission, and diffraction from those objects. This process can be repeated recursively. Rays shot from sound sources and rays shot from the listener can intersect, and the path along which they intersect is called a sound propagation path. Ultimately, a sound propagation path can refer to the effective path along which sound originating from a sound source travels through reflection, transmission, absorption, and diffraction to reach the listener. The final sound can be calculated using these sound propagation paths.

[0055]

[0056] Figure 3 is a drawing explaining mode switching of a conventional sound propagation unit.

[0057] Referring to FIG. 3, a conventional sound propagation unit (SPU) can operate sequentially according to DirectTrans, Reflection / Diffraction, and Reverb modes. The modes (SPU modes) of the sound propagation unit can be broadly divided into a sound path mode (S320) and a reverb mode (S350). The sound path mode (S320) can include a DirectTrans mode (S310) that processes direct sound and transmitted sound, and a Reflection / Diffraction mode (S330) that processes reflected sound and diffraction sound, and the Reverb mode (S350) can correspond to a mode that processes reverberant sound.

[0058] The DirectTrans mode (S310) may be a mode for determining direct sound and transmitted sound depending on whether an obstacle exists after conducting a test by creating a straight ray between the listener and the sound source. The Reflection / Diffraction mode (S330) may find an early reflection sound path. The early reflection sound path may include a reflection sound path and a diffraction sound path. The Reverb mode (S350) is performed at the very last stage and may find information for creating late reverberation sound.

[0059]

[0060] Figure 4 is a drawing explaining a conventional sound propagation processing process.

[0061] Referring to FIG. 4, the setup processing step (S410) may correspond to a step for controlling the switching of the mode (SPU mode) of the sound propagation unit and setting information necessary for generating guide ray information generated from a listener and reverb ray information generated from a sound source for a visibility test. After the setting is completed, depending on the mode (SPU mode) of the sound propagation unit, one of the guide ray or reverb ray information generated in the setup processing step (S410) itself and the reflection ray information generated through PPV (S450) may be selected and sent to the ray generation step (S420).

[0062] In the Ray Generation step (S420), a ray is generated based on the ray information generated in the setup processing step (S410), and the origin and direction of the ray can be calculated. The generated ray can be stored in the TnI input buffer, which is a space where input data for the traversal / intersection test is stored.

[0063] In the Traversal / Intersection test (TnI) step (S430), the ray generated from the ray generation step (S420) can be read from the TnI input buffer and checked for a triangle colliding with the ray in the acceleration structure. For the collision check, the TnI unit can repeatedly perform traversal and ray-triangle intersection tests on the acceleration structure. The determination result of the TnI unit can be stored in the TnI output buffer.

[0064] In the Hit Point Calculation step (S440), the intersection point between the ray and the geometry can be calculated based on the result transmitted from the TnI output buffer. If the ray is hit, triangle information data for the corresponding hit triangle ID can be requested. The calculated result and hit triangle information data can be transmitted to perform PPVnRGC (S450).

[0065] The PPVnRGC stage (S450) is a core functional block that performs sound propagation simulation, and can determine whether to perform a reflection sound path test or reverb geometry collection based on the ray characteristics of the received data. The PPVnRGC stage (S450) can be composed of a propagation path validator (PPV) and a reverb geometry collector (RGC).

[0066] The PPV step is the step that performs the search for direct, transmission, reflection, and diffraction sound paths. Typically, PPV uses the uniform theory diffraction method (UTD) to find the diffraction sound path, and the image source method to find the reflection sound path. If the direct sound path, transmission sound path, diffraction sound path, or reflection sound path is found, PPV can generate the information necessary for calculating the impulse response and pass it to the IR Calculator.

[0067] RGC can utilize reverb geometry information required for calculating reverb time. Reverb time is one of the important factors for generating natural reverb sound, and statistical acoustic models can generally be used to calculate reverb time. Among these, the Eyring model is a model that calculates energy decay within a single space and is widely used for its speed and simplicity. To calculate reverb time, the valid path triangle found through the propagation path test among the triangle information collided with the guide ray, and the reverb triangle, which is the hit triangle information found with the reverb ray, can be used.

[0068] At this time, since reverb cannot be calculated if the path triangles are not all generated, the sound path mode can be repeatedly performed until the path triangle information is all generated. Once the path triangle and reverb triangle information are prepared, sorting can be performed to efficiently and quickly process the removal of duplicate triangles from the path triangle and reverb triangle and the comparison of the path triangle and reverb triangle ID. The IDs of the sorted path triangle and reverb triangle can be compared to find triangles with the same ID and calculate valid reverberation information. The calculated valid reverberation data can be passed to the IR calculator.

[0069] Finally, in the IR calculation step (S460), the impulse response (IR) or reverb impulse response (reverb IR) of the valid direct / transmission, reflection, and diffraction paths processed from the PPVnRGC step (S450) can be calculated and stored in the valid path buffer. Once the path data is stored in the valid path buffer, sound propagation for the current frame is completed and can be sent to the auralization processing step.

[0070]

[0071] FIG. 5 is a drawing illustrating a real-time sound visualization system according to the present invention.

[0072] Referring to FIG. 5, a real-time sound visualization system (500) can be implemented including a user terminal (510), a sound visualization device (530), and a database (550), and can execute a real-time sound visualization operation based on sound tracing according to the present invention.

[0073] Specifically, the user terminal (510) may be a computing device operated by a user and capable of utilizing services related to real-time sound visualization via a network. The user terminal (510) may be implemented as a smartphone, laptop, or computer, but is not necessarily limited thereto, and may be implemented as various devices such as a tablet PC.

[0074] In particular, the user terminal (510) may correspond to a VR device that can experience sound paths visualized in a three-dimensional virtual space by executing a real-time sound visualization method based on sound tracing according to the present invention. In addition, the user terminal (510) may be implemented as one device constituting a real-time sound visualization system (100) according to the present invention, and the real-time sound visualization system (100) may be implemented by being modified in various forms according to the purpose of real-time sound visualization.

[0075] A user terminal (510) can be connected to an audio visualization device (530) via a network, and multiple user terminals (510) can be connected to the audio visualization device (530) simultaneously. The user terminal (510) can install and execute a dedicated program or application for interfacing with the audio visualization device (530).

[0076] The sound visualization device (530) may be implemented as a computer or server that performs a real-time sound visualization method based on sound tracing according to the present invention. Here, the real-time sound visualization method based on sound tracing according to the present invention may include a process of performing sound tracing for sound rendering and a process of performing visualization of sound paths.

[0077] In addition, the sound visualization device (530) can be connected to the user terminal (510) via a wired network or a wireless network such as Bluetooth, WiFi, LTE, etc., and can transmit and receive data with the user terminal (510) via the network. In addition, the sound visualization device (530) can be implemented to operate in connection with an independent external system (not shown in FIG. 1). For example, the sound visualization device (530) can be linked with a system that implements 3D virtual reality or augmented reality to provide the user with virtual reality or augmented reality through the user terminal (510).

[0078] In addition, in FIG. 1, the sound visualization device (530) is depicted as a device independent of the user terminal (510), but it is not necessarily limited thereto, and it can be implemented as a logical storage device included in the user terminal (510).

[0079] The database (550) may correspond to a storage device that stores various information required during the operation of the sound visualization device (530). The database (550) may store data for sound tracing and data for real-time sound visualization, but is not necessarily limited thereto, and may store information collected or processed in various forms during the process of the sound visualization device (530) performing the real-time sound visualization method based on sound tracing according to the present invention.

[0080] In addition, in FIG. 1, the database (550) is depicted as a device independent of the sound visualization device (530), but it is not necessarily limited thereto, and it can be implemented as a logical storage device included in the sound visualization device (530).

[0081]

[0082] Fig. 6 is a drawing explaining the system configuration of the acoustic visualization device of Fig. 5.

[0083] Referring to FIG. 6, the sound visualization device (530) may include a processor (610), a memory (630), a user input / output unit (650), and a network input / output unit (670).

[0084] The processor (610) can execute a real-time sound visualization procedure based on sound tracing according to an embodiment of the present invention, manage the memory (630) that is read or written in this process, and schedule a synchronization time between the volatile memory and the non-volatile memory in the memory (630). The processor (610) can control the overall operation of the sound visualization device (530), and is electrically connected to the memory (630), the user input / output unit (650), and the network input / output unit (670) to control the data flow therebetween. The processor (610) can be implemented as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) of the sound visualization device (530).

[0085] The memory (630) may include an auxiliary memory device implemented as a non-volatile memory such as an SSD (Solid State Disk) or an HDD (Hard Disk Drive) and used to store all data required for the sound visualization device (530), and may include a main memory device implemented as a volatile memory such as a RAM (Random Access Memory). In addition, the memory (630) may store a set of commands that execute a real-time sound visualization method based on sound tracing according to the present invention by being executed by an electrically connected processor (610).

[0086] The user input / output unit (650) includes an environment for receiving user input and an environment for outputting specific information to the user, and may include, for example, an input device including an adapter such as a touchpad, a touch screen, a virtual keyboard, or a pointing device, and an output device including an adapter such as a monitor or a touch screen. In one embodiment, the user input / output unit (650) may correspond to a computing device connected via a remote connection, in which case the sound visualization device (530) may be performed as an independent server.

[0087] The network input / output unit (670) provides a communication environment for connecting to a user terminal (510) via a network, and may include, for example, an adapter for communication such as a Local Area Network (LAN), a Metropolitan Area Network (MAN), a Wide Area Network (WAN), and a Value Added Network (VAN). In addition, the network input / output unit (670) may be implemented to provide a short-range communication function such as WiFi or Bluetooth, or a wireless communication function of 4G or higher for wireless transmission of data.

[0088]

[0089] Figure 7 is a drawing explaining the functional configuration of the processor of Figure 6.

[0090] Referring to FIG. 7, the sound visualization device (530) can perform a real-time sound visualization method based on sound tracing according to the present invention through a processor (610). To this end, the processor (610) can include a position tracking unit (710), a coordinate conversion unit (730), a sound path determination unit (750), a sound visualization unit (770), and a control unit (not shown in FIG. 7).

[0091] At this time, the embodiments of the present invention do not necessarily include all of the above-described components simultaneously. Depending on the embodiment, some of the above-described components may be omitted, or some or all of the above-described components may be selectively included. The operation of each component will be described in detail below.

[0092] The location tracking unit (710) can track the location information of the user and the sound source in real time in a three-dimensional space including at least the user and the sound source. Here, the three-dimensional space may correspond to a virtual audio space, and the virtual audio space may correspond to a three-dimensional acoustic environment implemented through a computer simulation. In addition, the user corresponds to a user connected to the virtual space, but in some cases, it may correspond to various objects moving in the space. That is, the location tracking unit (710) can track the location of the user in the three-dimensional space by tracking the head movement or hand movement of the user connected to the three-dimensional space through a device (e.g., a VR headset, etc.) worn by the user. The location tracking unit (710) can perform the location tracking operation by utilizing GPS, IMU, and a camera, etc. In addition, the location tracking unit (710) can analyze sounds generated from sound sources such as vocals, bass, and guitar to track the location of the sound source in the three-dimensional space.

[0093] In one embodiment, the location tracking unit (710) can track the location information of each of a plurality of users and sound sources in real time. That is, the location tracking unit (710) can perform a real-time location tracking operation for each of a plurality of users and a plurality of sound sources. The location tracking unit (710) can generally be implemented to track up to 16 users and 16 sound sources simultaneously, but is not necessarily limited thereto. The location tracking unit (710) can provide a richer and more accurate sound path based on the location information of various users and sound sources, and can provide each user with a vivid experience even in a real-time environment where multiple users participate simultaneously.

[0094] Meanwhile, the location tracking unit (710) can provide a user interface that allows the user to directly select a location tracking target through the user terminal (510), and the user can perform a location tracking operation by selecting a specific user or a specific sound source through the user interface.

[0095] The coordinate conversion unit (730) can receive location information of a user and a sound source and convert them into coordinates in a three-dimensional space. First, the coordinate conversion unit (730) can receive location information of the user and the sound source from the location tracking unit (710). In this case, the location information can include a user ID and location information for identifying the user, and a sound source ID and location information for identifying the sound source. In addition, the coordinate conversion unit (730) can calculate coordinate information corresponding to the locations of the user and the sound source through a coordinate conversion operation on a three-dimensional coordinate system whose origin is a reference point of the three-dimensional space. For example, the coordinate conversion unit (730) can generate x, y, and z coordinate information in a three-dimensional space based on the user's GPS coordinates received from the location tracking unit (710). Thereafter, the coordinate information converted by the coordinate conversion unit (730) can be transmitted to the sound path determination unit (740).

[0096] The sound path determination unit (750) can determine a candidate sound path that reaches the user among a plurality of sound propagation paths generated from a sound source based on spatial information of a three-dimensional space. The sound path determination unit (750) can generate and utilize a sound propagation model for generating a sound propagation path, and the sound propagation model can include a spatial model that implements a three-dimensional space based on information about the structure and characteristics of the three-dimensional space, objects arranged in the space (e.g., walls, furniture, obstacles, etc.), a sound source model that implements each sound source based on information about the location and directionality, frequency response, and intensity of the sound source, and an acoustic diffusion model that implements a process in which sound spreads in space.

[0097] Additionally, the sound path determination unit (750) can generate multiple sound propagation paths generated from each sound source through a sound propagation model. For example, the sound path determination unit (750) can generate random sound propagation paths that advance in all directions centered on each sound source. In this case, the sound path determination unit (750) can limit the number of sound propagation paths generated to a preset number.

[0098] Thereafter, the sound path determination unit (750) can select candidate sound paths by selecting paths that are highly likely to reach the user from among the generated sound propagation paths. In other words, the candidate sound path may correspond to a sound propagation path that can ultimately reach the user as a result of the sound generated from the sound source traveling along the path in the corresponding space. The sound path determination unit (750) can select candidate sound paths by simulating the sound propagation process in a three-dimensional space based on a sound propagation model and determining whether the simulation result reaches the user.

[0099] In one embodiment, the sound path determination unit (750) can classify a plurality of sound propagation paths according to sound types including direct sound, diffraction sound, and reflection sound. The sound types can be classified according to whether the propagating sound reaches the user directly from the sound source, reaches the user after being diffracted by a wall or an obstacle, or reaches the user after being reflected by a wall or an obstacle. The sound path determination unit (750) can classify which type of sound propagation path each sound propagation path belongs to, such as direct sound, diffraction sound, and reflection sound, through sound propagation modeling, and can group them into direct sound paths, diffraction sound paths, or reflection sound paths according to the classification results.

[0100] In one embodiment, the sound path determination unit (750) may determine candidate sound paths reaching the user, including physical properties of materials constituting the three-dimensional space. Here, the physical properties of the material may correspond to the degree of reflection, reflection-diffraction, refraction, and absorption of sound according to the material, and is not necessarily limited thereto, and may further include sound absorption rate, reflectance, and scattering coefficient, etc. The sound path determination unit (750) may analyze the material constituting the three-dimensional space, simulate the degree of reflection, reflection-diffraction, refraction, and absorption of sound in the three-dimensional space according to the physical properties of the material, and select candidate sound paths according to the simulation results.

[0101] The sound visualization unit (770) can visualize and display at least some of the candidate sound paths in a three-dimensional space. Through this, the user can intuitively understand how the sound is propagated in the surrounding environment. Specifically, the sound visualization unit (770) can receive candidate sound path data selected by the sound path determination unit (750) and visualize and display the sound path in a three-dimensional space implemented based on 3D spatial information. At this time, each path data may include the starting point (i.e., the location of the sound source) and the end point (i.e., the location of the user) of the path, the type and intensity of the path, etc. Various objects (e.g., walls, furniture, obstacles, etc.) that virtually implement the actual space may be placed in the three-dimensional space, and an avatar representing the user and a sound source object representing the sound source, etc. may be displayed.

[0102] In addition, the sound visualization unit (770) can apply various visualization methods for path visualization. For example, the sound visualization unit (770) can express each path by visualizing it as a line, and can adjust the color, thickness, transparency, and intensity of the line. In addition, the sound visualization unit (770) can express each path in the form of a tube or a particle, etc. The sound paths visualized in 3D space through the sound visualization unit (770) can be provided to the user through the user terminal (510). That is, the user can visually recognize the visualized sound paths while connected to a virtual 3D space through a VR headset. Meanwhile, the sound visualization unit (770) can provide a user interface through the user terminal (510) that allows the user to directly select a visualization method.

[0103] In one embodiment, the sound visualization unit (770) can visualize and display candidate sound paths centered on a specific user or sound source among multiple users and sound sources. The sound visualization unit (770) can provide a user interface via the user terminal (510), and a user can select one or more users through the user interface to selectively visualize only the sound paths reaching the user. In addition, the user can select a specific sound source in addition to the user, in which case only the sound paths generated from the corresponding sound source can be selectively visualized.

[0104] In addition, the sound visualization unit (770) can visualize candidate sound paths centered on a specific user or a specific sound source selected by the user, and can display the sound source or user associated with the visualized path so that it is visualized and distinguished from other users. For example, if users A, B, and C exist in a three-dimensional space and sound sources s1 and s2 exist, the sound visualization unit (770) can basically visualize sound paths that interconnect users A, B, and C and sound sources s1 and s2. At this time, if user A is selected and a sound path exists only between user A and sound source s2, the sound visualization unit (770) can visualize and display the corresponding sound path while highlighting sound source s2. That is, user A can visually recognize the path of the sound coming into him / her and at the same time easily detect the sound source of the corresponding sound among various sound sources.

[0105] In one embodiment, the sound visualization unit (770) can adjust the visualization conditions of candidate sound paths according to characteristic information of a specific user or a specific sound source. For example, the sound visualization unit (770) can highlight and display sound paths visualized within a field of view (FOV) set based on the user's head direction. As another example, the sound visualization unit (770) can adjust the visualization speed of sound paths according to the user's moving speed. That is, when the user moves quickly, the sound visualization unit (770) can increase the visualization speed of the sound path so that the user can track the sound path more easily. As another example, the sound visualization unit (770) can adjust the line thickness of the sound path according to the frequency characteristics of the sound source. That is, the sound visualization unit (770) can visualize a low-frequency sound source with a thick line and a high-frequency sound source with a thin line. In addition, the sound visualization unit (770) can visualize sound paths by adjusting the visualization conditions according to the directionality or intensity of the sound source.

[0106] In one embodiment, the sound visualization unit (770) can visualize candidate sound paths by distinguishing colors according to sound type. For example, the sound visualization unit (770) can visualize direct sound paths in red, diffraction sound paths in light blue, and reflected sound paths in blue. The sound visualization unit (770) can selectively visualize only specific sound types or visualize them by emphasizing them so that they are distinguished from other sound types.

[0107] In one embodiment, the sound visualization unit (770) can visualize and display candidate sound paths in an actual space corresponding to a three-dimensional space using an augmented reality method. For example, a user can capture a picture of an actual space using a smartphone or tablet camera, and the sound visualization unit (770) can display the candidate sound paths by overlaying them in the form of virtual objects on the actual captured image. As another example, a user can directly view the actual space while wearing an AR headset and experience an augmented reality experience in which the sound visualization unit (770) visualizes and displays the candidate sound paths.

[0108] In one embodiment, the sound visualization unit (770) may be implemented as a configuration for augmented reality visualization, including multiple independent modules. This will be described in more detail with reference to FIG. 8.

[0109] The control unit (not shown in FIG. 7) controls the overall operation of the sound visualization device (530) and can manage the control flow or data flow between the position tracking unit (710), the coordinate conversion unit (730), the sound path determination unit (750), and the sound visualization unit (770).

[0110]

[0111] Figure 8 is a drawing illustrating one embodiment of an acoustic visualization unit according to the present invention.

[0112] Referring to FIG. 8, the sound visualization device (530) can visualize and display at least some of the candidate sound paths in a three-dimensional space through the sound visualization unit (770), and can visualize and display the candidate sound paths in an augmented reality manner in an actual space corresponding to the three-dimensional space. In particular, the sound visualization unit (770) can include an image recognition module (810), a SLAM (Simultaneous Localization And Mapping) module (830), a space matching module (850), and a graphic augmentation module (870) as a configuration for visualization in an augmented reality manner.

[0113] The image recognition module (810) can perform an operation of receiving and recognizing a real space image captured on a user terminal (510), such as a smartphone, tablet, or AR headset. The image recognition module (810) can extract feature data regarding feature points, patterns, and objects from the real space image, and can recognize and classify the real space image based on the extracted feature data.

[0114] The SLAM module (830) can perform operations to track the position and direction of a camera in a dynamic environment while simultaneously constructing a virtual three-dimensional space. For example, the SLAM module (830) can simultaneously estimate a camera trajectory and a three-dimensional space based on a camera image sequence using a SLAM algorithm. In particular, the SLAM module (830) can perform more accurate estimation operations by utilizing other sensor data, such as an acceleration sensor, a gyroscope, and LiDAR, in addition to the camera sensor.

[0115] The spatial matching module (850) can perform an operation of extracting feature points from a real space and matching them with a virtual three-dimensional space. The spatial matching module (850) can match feature points of a real space with feature points of a three-dimensional space to maintain consistency between the two spaces. For example, the spatial matching module (850) can extract feature points from a real space image using a feature point extraction algorithm. In addition, the spatial matching module (850) can extract feature points based on points, lines, or planes that constitute a virtual three-dimensional space. Thereafter, the spatial matching module (850) can perform a feature point matching operation based on the similarity between feature points, and can perform an operation of converting the virtual three-dimensional space to fit the real space based on the matched feature points. If a visualized sound path exists in the three-dimensional space, the spatial matching module (850) can additionally perform an operation of adjusting the sound path based on the converted three-dimensional space.

[0116] The graphic augmentation module (870) can perform an operation of creating a virtual object and overlaying it on a real space image to implement an augmented image. The graphic augmentation module (870) can use a pre-built virtual object or model a virtual object, render it, and then overlay it on the real space image to display it. In this case, the graphic augmentation module (870) can implement a natural visualization by adjusting the position, direction, and lighting of the virtual object overlaid on the real space image.

[0117]

[0118] Figure 9 is a flowchart illustrating a real-time sound visualization method based on sound tracing according to the present invention.

[0119] Referring to FIG. 9, the sound visualization device (530) can track the location information of the user and the sound source in real time in a three-dimensional space including at least the user and the sound source through the processor (610) (step S910). The sound visualization device (530) can receive the location information of the user and the sound source through the processor (610) and convert it into coordinates in the three-dimensional space (step S930).

[0120] In addition, the sound visualization device (530) can determine a candidate sound path reaching the user among a plurality of sound propagation paths generated from a sound source based on spatial information of a three-dimensional space through the processor (610) (step S950). The sound visualization device (530) can visualize and display at least some of the candidate sound paths in a three-dimensional space through the processor (610) (step S970).

[0121]

[0122] Although the present invention has been described above with reference to preferred embodiments thereof, it will be understood by those skilled in the art that various modifications and changes may be made to the present invention without departing from the spirit and scope of the present invention as set forth in the claims below.

[0123]

[0124] [National Research and Development Project Supporting This Invention]

[0125] [Project ID] 1425174979

[0126] [Assignment Number] S3317348 / RS-2022-TI024238

[0127] [Ministry Name] Ministry of SMEs and Startups

[0128] [Name of Project Management (Specialist) Agency] Small and Medium Business Technology Information Promotion Agency

[0129] [Research Project Name] Startup Growth Technology Development

[0130] [Research Project Title] Development of a Sound Rendering Semiconductor IP Supporting Dynamic Diffraction for Real-Time 3D Audio in a Hyper-Realistic Metaverse

[0131] [Name of the project performing organization] Exarion Co., Ltd.

[0132] Research Period: September 1, 2022 - August 31, 2024

[0133]

[0134] [Explanation of symbols]

[0135] 500: Real-time sound visualization system

[0136] 510: User terminal 530: Sound visualization device

[0137] 550: Database

[0138] 610: Processor 630: Memory

[0139] 650: User I / O section 670: Network I / O section

[0140] 710: Position tracking unit 730: Coordinate conversion unit

[0141] 750: Sound path determination unit 770: Sound visualization unit

[0142] 810: Image recognition module 830: SLAM module

[0143] 850: Spatial Matching Module 870: Graphics Augmentation Module

Claims

1. A location tracking unit that tracks location information of a user and a sound source in real time in a three-dimensional space including at least a user and a sound source; A coordinate conversion unit that receives the location information of the user and sound source and converts it into coordinates in the three-dimensional space; A sound path determination unit that determines a candidate sound path that reaches the user among a plurality of sound propagation paths generated from the sound source based on spatial information of the three-dimensional space including the physical properties of the materials constituting the three-dimensional space; and A real-time sound visualization device based on sound tracing, comprising: an acoustic visualization unit for visualizing and displaying at least a portion of the candidate sound paths in the three-dimensional space.

2. In the first paragraph, the position tracking unit A real-time sound visualization device based on sound tracing, characterized by tracking the location information of multiple users and sound sources in real time.

3. In the first paragraph, the sound path determining unit A real-time sound visualization device based on sound tracing, characterized in that the plurality of sound propagation paths are classified according to sound types including direct sound, diffraction sound, and reflection sound.

4. In the second paragraph, the sound visualization unit A real-time sound visualization device based on sound tracing, characterized in that it visualizes and displays the candidate sound path centered on a specific user or a specific sound source among the plurality of users and sound sources.

5. In the fourth paragraph, the sound visualization unit A real-time sound visualization device based on sound tracing, characterized in that it adjusts the visualization conditions of the candidate sound path according to the characteristic information of the specific user or the specific sound source.

6. In the first paragraph, the sound visualization unit A real-time sound visualization device based on sound tracing, characterized in that the candidate sound path is visualized by distinguishing colors according to sound type.

7. In the first paragraph, the sound visualization unit A real-time sound visualization device based on sound tracing, characterized in that it visualizes and displays the candidate sound path in an augmented reality manner in an actual space corresponding to the three-dimensional space.

Images