Device for three-dimensional sound reproduction in virtual sound propagation space

Abstract

The present invention relates to a device for three-dimensional sound reproduction in a virtual sound propagation space, the device comprising: a virtual sound propagation unit for determining at least one sound success path from the location of a sound source generating an original sound of N channels (N is a natural number) to the location of a sound sink in a virtual sound propagation space; an adaptive sound path buffering unit for buffering the at least one sound success path on the basis of a specific criterion; and a 3D virtual sound auralization unit for generating a sound output of M channels (M is a natural number) on the basis of channel attributes of the original sound, reachability attributes of the at least one sound success path, and material attributes of the virtual sound propagation space.

Description

3D sound reproduction device in a virtual sound propagation space

[0001] The present invention relates to a three-dimensional sound reproduction technology, and more specifically, to a three-dimensional sound reproduction device in a virtual sound propagation space capable of generating 3D sound output by optimizing a sound success path in a virtual sound propagation space.

[0002]

[0003] Recently, interest in virtual reality technology has been rapidly increasing due to advancements in mobile technology, graphics technology, and sensory input / output technology. To support a realistic virtual reality environment, it is necessary to reproduce not only visual spatiality through virtual space but also high-quality auditory spatiality. To reproduce auditory spatiality, multi-channel audio systems or 3D sound technology using Head Related Transfer Functions (HRTF) can be used.

[0004] However, multi-channel audio used for 3D sound reproduction presents challenges, such as the need for a dedicated speaker system and space for its installation. Most 3D sound technologies based on head transfer functions utilize pre-calculated acoustic control filters or reproduce auditory spatiality within simple virtual spaces, such as rectangular shoe boxes. Since physical effects regarding the surrounding environment and the materials of complex objects in the virtual space are not reflected, these technologies have limitations in reproducing realistic sound.

[0005] To address these limitations, 3D sound technologies based on geometric or numeric methods are being introduced. Among the various geometric methods, a method that combines ray tracing technology in 3D graphics with sound processing technology is called sound tracing. Sound tracing is a type of sound rendering technique that generates sound by tracing the sound propagation paths between a listener and a sound source.

[0006] Sound tracing can generate realistic sound sources by creating various sound propagation paths, such as direct, reflection, and edge-diffraction paths, to reproduce realistic sound, but these sound propagation paths have an unstable problem when obstacles are newly created or removed between the sound source and the listener.

[0007]

[0008] [Prior Art Literature]

[0009] [Patent Literature]

[0010] Korean Registered Patent No. 10-1828908 (February 7, 2018)

[0011]

[0012] One embodiment of the present invention aims to provide a three-dimensional sound reproduction device in a virtual sound propagation space capable of generating 3D sound output by optimizing the sound success path in the virtual sound propagation space.

[0013] One embodiment of the present invention aims to provide a three-dimensional sound reproduction device capable of controlling the number of sound success paths from a sound sink generating an N-channel original sound in a virtual sound propagation space to a sound sink generating an M-channel sound output.

[0014] One embodiment of the present invention aims to provide a three-dimensional sound reproduction device in a virtual sound propagation space capable of selecting and buffering a sound success path that has reached a sound sink from the time of occurrence of the original sound up to a specific reference time.

[0015] One embodiment of the present invention aims to provide a three-dimensional sound reproduction device in a virtual sound propagation space capable of creating a new virtual channel in the original sound based on the location of each channel constituting the channel attributes of the original sound, the mixing coherence of each channel, and the panning distribution of each channel.

[0016]

[0017] Among the embodiments, the virtual sound propagation unit determines at least one sound success path from the location of a sound source to the location of a sound sink in a virtual sound propagation space, and the sound source includes a virtual sound propagation unit that generates an original sound of N (where N is a natural number) channels, an adaptive sound path buffering unit that buffers the at least one sound success path according to a specific criterion, and a 3D virtual sound auralization unit that generates a 3D virtual sound output of M (where M is a natural number) channels at the sound sink based on the channel properties of the original sound, the arrival properties of the at least one sound success path, and the material properties of the virtual sound propagation space.

[0018] The virtual sound propagation unit can optimize the number of at least one sound success path based on the location of the sound sink in the virtual sound propagation space.

[0019] The virtual sound propagation unit can determine a minimum value for the number of at least one sound success path by analyzing the spatial attributes of the virtual sound propagation space.

[0020] The adaptive sound path buffering unit above can select and buffer sound success paths that have reached the sound sink from the time of occurrence of the original sound up to a specific reference time.

[0021] The above 3D virtual sound auralization unit can create a new virtual channel in the original sound based on the location of each channel constituting the channel attributes of the original sound, the mixing coherence of each channel, and the panning distribution of each channel.

[0022] The above 3D virtual sound auralization unit analyzes the time difference of each channel constituting the arrival attribute of the at least one sound success path, and if the time difference is greater than a certain standard, it can delete a specific channel in the original sound.

[0023] The above 3D virtual sound auralization unit can amplify a specific channel in the original sound by analyzing the material properties of the virtual sound propagation space that constitute the sound reflectivity of static obstacles and the sound absorption of dynamic obstacles placed in each of the at least one sound success path.

[0024]

[0025] The disclosed technology may have the following effects. However, this does not mean that a specific embodiment must include all of the following effects or only the following effects; therefore, the scope of the rights of the disclosed technology should not be understood as being limited by this.

[0026] A three-dimensional sound reproduction device in a virtual sound propagation space according to one embodiment of the present invention can generate a 3D sound output by optimizing the sound success path in the virtual sound propagation space.

[0027] A three-dimensional sound reproduction device in a virtual sound propagation space according to one embodiment of the present invention can adjust the number of sound success paths from a sound sink generating an N-channel original sound in the virtual sound propagation space to a sound sink generating an M-channel sound output.

[0028] A three-dimensional sound reproduction device in a virtual sound propagation space according to one embodiment of the present invention can select and buffer a sound success path that reached the sound sync from the time of occurrence of the original sound until a specific reference time.

[0029] A three-dimensional sound reproduction device in a virtual sound propagation space according to one embodiment of the present invention can create a new virtual channel in the original sound based on the location of each channel constituting the channel attributes of the original sound, the mixing coherence of each channel, and the panning distribution of each channel.

[0030]

[0031] FIG. 1 is a drawing illustrating a three-dimensional sound reproduction system according to the present invention.

[0032] Figure 2 is a diagram illustrating the system configuration of the three-dimensional sound reproduction device of Figure 1.

[0033] Figure 3 is a diagram illustrating the configuration of the three-dimensional sound reproduction device of Figure 1.

[0034] FIG. 4 is a flowchart illustrating the functional configuration of a three-dimensional sound reproduction device according to the present invention.

[0035] Figure 5 is a diagram illustrating the virtual sound propagation unit in Figure 3.

[0036] FIG. 6 is a flowchart illustrating the process of generating an M-channel sound output by inputting an N-channel original sound in a three-dimensional sound reproduction device according to the present invention.

[0037]

[0038] The description of the present invention is merely an example for structural or functional explanation, and therefore the scope of the present invention should not be interpreted as being limited by the examples described in the text. That is, since the examples are subject to various modifications and may take various forms, the scope of the present invention should be understood to include equivalents capable of realizing the technical concept. Furthermore, the objectives or effects presented in the present invention do not imply that a specific example must include all of them or only such effects; therefore, the scope of the present invention should not be understood as being limited by them.

[0039] Meanwhile, the meaning of the terms described in this application should be understood as follows.

[0040] Terms such as "first," "second," etc., are intended to distinguish one component from another, and the scope of rights shall not be limited by these terms. For example, the first component may be named the second component, and similarly, the second component may be named the first component.

[0041] When it is stated that one component is "connected" to another component, it should be understood that it may be directly connected to that other component, or that there may be other components in between. Conversely, when it is stated that one component is "directly connected" to another component, it should be understood that there are no other components in between. Meanwhile, other expressions describing the relationships between components, such as "between" and "exactly between," or "adjacent to" and "directly adjacent to," should be interpreted in the same way.

[0042] A singular expression should be understood to include a plural expression unless the context clearly indicates otherwise, and terms such as "include" or "have" are intended to specify the existence of the implemented features, numbers, steps, actions, components, parts, or combinations thereof, and should be understood not to preclude the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0043] In each step, identifiers (e.g., a, b, c, etc.) are used for convenience of explanation and do not describe the order of the steps; the steps may occur differently from the specified order unless a specific order is clearly indicated in the context. That is, the steps may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the reverse order.

[0044] The present invention may be implemented as computer-readable code on a computer-readable recording medium, and the computer-readable recording medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. Additionally, the computer-readable recording medium may be distributed across networked computer systems, so that computer-readable code can be stored and executed in a distributed manner.

[0045] Unless otherwise defined, all terms used herein have the same meaning as generally understood by those skilled in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted as having meanings consistent with the context of the relevant technology and should not be interpreted as having an ideal or overly formal meaning unless explicitly defined in this application.

[0046]

[0047] FIG. 1 is a drawing illustrating a three-dimensional sound reproduction system according to the present invention.

[0048] Referring to FIG. 1, a three-dimensional sound regeneration system (100) may be implemented including a 3D sound space (110), a three-dimensional sound regeneration device (130), and a database (150), and may perform a three-dimensional sound regeneration operation.

[0049] The 3D sound space (110) corresponds to an actual physical sound space that supports the implementation of a virtual sound propagation space through a three-dimensional sound reproduction device (130). For example, the 3D sound space (110) can be implemented as a space in a plaza area, such as a movie theater. As another example, the 3D sound space (110) can be implemented as a space in a private area, such as a user's headset.

[0050] The 3D sound space (110) can physically fix or variably change the location of a sound source, such as a speaker. For example, the 3D sound space (110) can physically fix the location of the sound source when occupied by a large number of people (i.e., a large number of sound sinks). For another example, the 3D sound space (110) can variably change the location of the sound source when utilized by a single user (i.e., a single sound sink).

[0051] The 3D sound space (110) may be composed of a singular or multiple units, and if composed of multiple units, it may be composed of a first 3D sound space (e.g., 110a), a second 3D sound space (e.g., 110b), ..., an nth (n is a natural number) 3D sound space that can be independently controlled by a 3D sound reproduction device (130).

[0052] A three-dimensional sound reproduction device (130) may be implemented as a computing device (e.g., a server) that performs a three-dimensional sound reproduction method according to one embodiment of the present invention and is implemented as a computing program. The three-dimensional sound reproduction device (130) can play a sound source in a virtual space in real time by calculating a sound success path in real time that corresponds to a path for transmitting a sound generated from the location of a sound source to the location of a sound sink, and can allow a sound sink (e.g., a listener of 110a or 110b) to experience the three-dimensional sound effect of a sound source played in a non-existent virtual space in a real space.

[0053] A three-dimensional sound reproduction device (130) can transmit sound to a sound sink (e.g., a listener of 110a or 110b) in the three-dimensional sound space (110) by calculating an impulse response in real time through a sound source (e.g., a sound source or speaker of 110a or 110b) in the three-dimensional sound space (110) and taking into account a statically fixed or dynamically moving obstacle in the three-dimensional sound space (110).

[0054] The three-dimensional sound reproduction device (130) can transmit and receive data with the sound source (e.g., a sound source or speaker of 110a or 110b) by connecting to the sound source of the 3D sound space (110) via a wired network or a wireless network such as Bluetooth, WiFi, or LTE.

[0055] The 3D sound regeneration device (130) can control the 3D sound space (110) independently or dependently, and can determine independent or dependent control based on the correlation of sound sinks. Here, the correlation of sound sinks may correspond to acoustic connectivity based on the location where the sound sinks are located. For example, the 3D sound regeneration device (130) can deliver the same sound to each sound sink by dependently controlling at least one sound sink located in the 3D sound space (110), such as a movie theater. Additionally, the 3D sound regeneration device (130) can control the 3D sound space (110) independently in the case of an environment where different sound effects need to be provided.

[0056] The database (150) may correspond to a storage device that stores sound source content of the three-dimensional sound regeneration device (130). The database (150) may provide sound source content to be delivered to the sound sink in response to a request for sound source content from the three-dimensional sound regeneration device (130). Although the database (150) is depicted as a device independent of the three-dimensional sound regeneration device (130), it is not necessarily limited thereto and can, of course, be implemented by being included in the three-dimensional sound regeneration device (130).

[0057]

[0058] Figure 2 is a diagram illustrating the system configuration of the three-dimensional sound reproduction device of Figure 1.

[0059] Referring to FIG. 2, the three-dimensional sound reproduction device (130) may include a processor (210), memory (230), user input / output unit (250), network input / output unit (270), and communication port unit (290).

[0060] The processor (210) can execute a three-dimensional sound regeneration service procedure according to an embodiment of the present invention, manage memory (230) that is read or written during this process, and schedule the synchronization time between volatile memory and non-volatile memory in memory (230). The processor (210) can control the overall operation of the three-dimensional sound regeneration device (130) and is electrically connected to memory (230), user input / output unit (250), and network input / output unit (270) to control the data flow between them. The processor (210) can be implemented as a CPU (Central Processing Unit) or GPU (Graphics Processing Unit) of the three-dimensional sound regeneration device (130).

[0061] The memory (230) may include an auxiliary storage device implemented as non-volatile memory such as an SSD (Solid State Disk) or HDD (Hard Disk Drive) and used to store all data required for the three-dimensional sound regeneration device (130), and may include a main memory device implemented as volatile memory such as RAM (Random Access Memory). Additionally, the memory (230) may store a set of instructions that execute a three-dimensional sound regeneration method according to an embodiment of the present invention by being executed by an electrically connected processor (210).

[0062] The user input / output unit (250) includes an environment for receiving user input and an environment for outputting specific information to the user, and may include an input device including an adapter such as a touch pad, touch screen, virtual keyboard, or pointing device, and an output device including an adapter such as a monitor or touch screen. In one embodiment, the user input / output unit (250) may correspond to a computing device connected via remote access, and in such case, the three-dimensional sound reproduction device (130) may be performed as an independent server.

[0063] The network input / output unit (270) provides a communication environment for connecting to a specific terminal through a network and may include an adapter for communication such as a LAN (Local Area Network), MAN (Metropolitan Area Network), WAN (Wide Area Network), and VAN (Value Added Network). Additionally, the network input / output unit (270) may be implemented to provide short-range communication functions such as WiFi and Bluetooth, or wireless communication functions of 4G or higher, for wireless transmission of learning data.

[0064] The communication port (290) is a hardware interface for connecting to external hardware, for example, the external hardware may include a printer, a mouse, and USB hardware. The communication port (290) can detect the connection of specific USB hardware and perform the role of a three-dimensional sound reproduction device (130).

[0065]

[0066] Figure 3 is a diagram illustrating the configuration of the three-dimensional sound reproduction device of Figure 1.

[0067] Referring to FIG. 3, the three-dimensional sound reproduction device (130) may include a virtual sound propagation unit (310), an adaptive sound path buffering unit (330), a 3D virtual sound auralization unit (350), and a control unit (370).

[0068] The three-dimensional sound reproduction device (130) does not necessarily have to include all of the above functional configurations simultaneously, and depending on each embodiment, it may be implemented by omitting some of the above configurations or by selectively including some or all of the above configurations. Additionally, the three-dimensional sound reproduction device (130) may be implemented as an independent module that selectively includes some of the above configurations, and the three-dimensional sound reproduction method according to the present invention may be performed through the interaction between each module. The operation of each configuration will be described in detail below.

[0069]

[0070] The virtual sound propagation unit (310) determines at least one sound success path from the location of a sound source to the location of a sound sink in a virtual sound propagation space, and the sound source can generate original sound of N (where N is a natural number) channels. Here, the sound source may correspond to a point where sound occurs, and may include, for example, a point where a gunshot occurs in a game and a point where a specific person speaks in a virtual meeting. In addition, the sound source may include various characteristics such as the location where sound occurs, the intensity of sound, directionality, and frequency band. The sound sink may correspond to a listener who is the subject listening to the sound generated from the sound source. In addition, the sound success path may correspond to the path of the sound that actually reached the sound sink among the entire sound propagation path.

[0071] Additionally, the virtual sound propagation unit (310) can determine at least one sound success path from the location of the sound source to the location of the sound sink by shooting a ray at the respective locations of the sound source and the sound sink. Here, the virtual sound propagation unit (310) can generate rays corresponding to reflection, transmission, and diffraction for a ray that collides with an obstacle in the ray shooting path during the process of shooting the ray, and this process can be performed recursively. The virtual sound propagation unit (310) can analyze the path of each shot ray and determine the ray that meets the ray shot from the sound source and the ray shot from the sound sink as the sound success path. That is, the virtual sound propagation unit (310) can generate a valid path through which sound originating from the sound source location arrives at the sound sink after undergoing reflection, transmission, absorption, diffraction, etc.

[0072] In one embodiment, the virtual sound propagation unit (310) can determine at least one sound success path and generate N-channel original sound from a sound source. Here, the virtual sound propagation unit (310) can implement a surround sound system in the process of generating N-channel original sound from a sound source. For example, the virtual sound propagation unit (310) can implement a surround sound system that provides sound having front, center, surround, and rear directions by implementing a surround sound system including 5.1 channels and 7.1 channels from a sound source.

[0073] In one embodiment, the virtual sound propagation unit (310) may classify sound success paths into direct paths and indirect paths and determine the importance of each sound success path. Here, the direct path may correspond to a path in which sound is transmitted directly between a sound source and a sound sink without any obstacles. The indirect path may correspond to a reflection path in which sound reaches a listener after colliding with an obstacle and reflecting, or a transmission path in which sound is transmitted to a sound sink by passing through an obstacle when there is an obstacle between a sound source and a sound sink. In one embodiment, the virtual sound propagation unit (310) may determine the importance of each sound success path and exclude sound success paths with low importance. For example, the virtual sound propagation unit (310) may determine the importance of sound success paths based on the sound quality and sound volume of the sound success paths and exclude sound success paths having sound quality and sound volume below a certain standard. The virtual sound propagation unit (310) is not necessarily limited to this and determines the importance of the sound success path according to the delay time, reflection and absorption characteristics of the sound success path, and can exclude sound success paths having a delay time of a certain level or more, sound success paths having a sound reflection count of a certain level or more, and sound success paths having an absorption rate of a certain level or more.

[0074] In one embodiment, the virtual sound propagation unit (310) can optimize the number of at least one sound success path based on the location of a sound sink in the virtual sound propagation space. Here, the virtual sound propagation unit (310) can optimize the sound success path by limiting the number of sound success paths according to the location of the sound sink. For example, the virtual sound propagation unit (310) can optimize the sound success path by calculating the distance from the sound source to the sound sink and preferentially selecting the sound success path having the minimum distance.

[0075] Additionally, the virtual sound propagation unit (310) can optimize the sound success path by selecting a sound success path that has an optimal reflection path according to the characteristics of the virtual sound propagation space. For example, if the virtual sound propagation space corresponds to a space having multiple reflection paths, such as a concert hall, the virtual sound propagation unit (310) can optimize the sound success path by prioritizing the selection of a sound success path that has sound directionality and sound quality above a certain level.

[0076] In one embodiment, the virtual sound propagation unit (310) can determine a minimum value for the number of at least one sound success path by analyzing the spatial attributes of the virtual sound propagation space. Here, the spatial attributes may correspond to the arrangement structure of static and dynamic obstacles within the virtual sound propagation space. The virtual sound propagation unit (310) can perform an analysis of the spatial attributes of the virtual sound propagation space and calculate the total sound success path according to the arrangement structure of static and dynamic obstacles within the virtual sound propagation space. Here, the virtual sound propagation unit (310) can select a sound success path having an importance level greater than a certain standard among the total sound success paths and determine a minimum value for the number of sound success paths capable of providing sound of a certain quality or higher.

[0077] Additionally, the virtual sound propagation unit (310) can detect changes in the sound success path according to the arrangement structure of static and dynamic obstacles within the virtual sound propagation space. For example, the virtual sound propagation unit (310) can detect changes in the sound success path due to the movement of dynamic obstacles and change the importance of the sound success path according to the movement of dynamic obstacles.

[0078] The adaptive sound path buffering unit (330) can buffer at least one sound success path based on specific criteria. Here, buffering may correspond to storing sound data to adjust for delays, reflections, and acoustic distortions occurring in the sound success path. Additionally, the specific criteria may correspond to reference values ​​for applying buffering, for example, time delay, sound quality, and phase. By performing buffering based on specific criteria for at least one sound success path, the adaptive sound path buffering unit (330) can temporarily store the path and process it appropriately at the required time. For example, the adaptive sound path buffering unit (330) can perform time synchronization of the sound success path by performing buffering on the sound success path where a time delay occurs when a time delay occurs between the sound success path corresponding to the direct path and the sound success path corresponding to the indirect path.

[0079] In one embodiment, the adaptive sound path buffering unit (330) can select and buffer sound success paths that have reached the sound sink from the time of occurrence of the original sound up to a specific reference time. Here, the adaptive sound path buffering unit (330) can store the sound success paths that have reached the sound sink from the time of occurrence of the original sound in a chronological order in the database (150). The adaptive sound path buffering unit (330) can exclude sound success paths that reach the sound sink after a specific reference time and perform buffering on sound success paths that reach the sound sink up to a specific reference time. Through this, the adaptive sound path buffering unit (330) can prevent distortion of the sound success paths by excluding sound success paths after a specific reference time when multiple sound success paths occur.

[0080] The 3D virtual sound auralization unit (350) can generate a 3D virtual sound output of M (M is a natural number) channels based on the channel properties of the original sound, the arrival properties of at least one sound success path, and the material properties of the virtual sound propagation space. Here, auralization may correspond to a technology that simulates the characteristics of sound through a computer and reproduces them audibly, for example, a technology that calculates how the sound will be heard within a specific space. In addition, the arrival properties may correspond to the distance, propagation time, sound pressure level, reflection, and absorption of the sound, and are not necessarily limited to these but may correspond to physical properties occurring in the path from the sound source to the sound sink. The material properties may correspond to the materials of the sound source, sound sink, and obstacles within the virtual sound propagation space. The 3D virtual sound auralization unit (350) can generate sound outputs of mono, stereo, and surround channels based on the arrival properties of the sound success path and the material properties of the virtual sound propagation space.

[0081] In one embodiment, the 3D virtual sound auralization unit (350) can create new virtual channels in the original sound based on the position of each channel constituting the channel attributes of the original sound, the mixing coherence of each channel, and the panning distribution of each channel. Here, the channel attributes may include the number of channels, channel positions, channel mixing, sampling rate, and bit depth, but are not necessarily limited thereto. The 3D virtual sound auralization unit (350) can create new virtual channels according to the position of the channels. Here, the 3D virtual sound auralization unit (350) can create new virtual channels including positions such as left, right, front left and right, rear left and right, and center.

[0082] Additionally, the 3D virtual sound auralization unit (350) can form a new virtual channel in the original sound based on the mixing consistency of each channel. Here, the mixing consistency may correspond to the degree of harmony of the phase relationship and interaction between each channel. The 3D virtual sound auralization unit (350) can analyze the phase relationship of each channel and determine whether sound cancellation or sound amplification occurs in a specific frequency band. That is, the 3D virtual sound auralization unit (350) can create a new virtual channel that can play the original sound simultaneously without a time difference based on the mixing consistency of each channel. In one embodiment, the 3D virtual sound auralization unit (350) can analyze the spatial correlation for each channel sound and create a new virtual channel having mixing consistency. That is, the 3D virtual sound auralization unit (350) can create a new virtual channel that provides sound generated in the same environment by adjusting the spatial correlation for each channel sound.

[0083] In one embodiment, the 3D virtual sound auralization unit (350) can create a new virtual channel in the original sound based on a panning distribution. Here, panning may correspond to a process of determining the position of the channel sound within the virtual sound propagation space. The 3D virtual sound auralization unit (350) can create a spatial image of the virtual sound propagation space by creating a new virtual channel in the original sound through the panning distribution. For example, the 3D virtual sound auralization unit (350) can provide a three-dimensional sound to the user by creating a virtual channel that provides a specific instrument sound through the panning distribution and placing it in the virtual sound propagation space.

[0084] In one embodiment, the 3D virtual sound auralization unit (350) analyzes the time difference of each channel constituting the arrival attribute of at least one sound success path, and if the time difference is greater than a specific standard, it can delete a specific channel in the original sound. Here, the 3D virtual sound auralization unit (350) can determine the average value between the channel having the minimum time and the channel having the maximum time based on a specific standard during the process of analyzing the time difference of each channel. The 3D virtual sound auralization unit (350) analyzes the time difference of each channel based on the average value of the channels and can remove the corresponding channel if the time difference is greater than a specific standard. Through this, the 3D virtual sound auralization unit (350) can minimize acoustic distortion caused by unnecessary delayed sound or reflected sound.

[0085] In one embodiment, the 3D virtual sound auralization unit (350) can amplify a specific channel in the original sound by analyzing the material properties of the virtual sound propagation space, which constitute the sound reflectance of a static obstacle and the sound absorption of a dynamic obstacle placed in each of at least one sound success path. Here, the sound reflectance may correspond to the ratio of sound reflected as the sound collides with the obstacle, and may be expressed as a value between 0 and 1, for example. Additionally, the sound absorption may correspond to the ratio of sound absorbed by the obstacle as the sound collides with the obstacle, and may correspond to a complementary concept to the sound reflectance. The 3D virtual sound auralization unit (350) can amplify a specific channel in the original sound by assigning sound reflectance and sound absorption according to the material of the static obstacle and the dynamic obstacle. For example, the 3D virtual sound auralization unit (350) can amplify specific channels and form echo and reverb spaces by forming a virtual sound propagation space with high reflectivity according to the material of static obstacles and dynamic obstacles.

[0086] The control unit (370) controls the overall operation of the 3D sound reproduction device (130) and can manage the control flow or data flow between the virtual sound propagation unit (310), the adaptive sound path buffering unit (330), and the 3D virtual sound auralization unit (350).

[0087]

[0088] FIG. 4 is a flowchart illustrating the functional configuration of a three-dimensional sound reproduction device according to the present invention.

[0089] Referring to FIG. 4, the three-dimensional sound regeneration device (130) determines at least one sound success path from the location of a sound source to the location of a sound sink in a virtual sound propagation space based on a virtual sound propagation unit (310), and the sound source can generate N-channel original sound (step S410). The three-dimensional sound regeneration device (130) can buffer at least one sound success path according to a specific standard through an adaptive sound path buffering unit (330) (step S430).

[0090] The 3D sound reproduction device (130) can generate an M-channel 3D virtual sound output from a sound sink based on the channel properties of the original sound, the arrival properties of at least one sound success path, and the material properties of the virtual sound propagation space through the 3D virtual sound auralization unit (350) (step S450).

[0091]

[0092] Figure 5 is a diagram illustrating the virtual sound propagation unit in Figure 3.

[0093] In FIG. 5, the virtual sound propagation unit (310) can generate a sound propagation path according to the properties of the obstacle. Here, the virtual sound propagation unit (310) can generate a sound propagation path by distinguishing it according to the movement properties of the obstacle and the material properties of the obstacle. For example, the virtual sound propagation unit (310) can classify the obstacle into static obstacles, dynamic obstacles, and quasi-static obstacles according to the movement properties of the obstacle. Here, a quasi-static obstacle may correspond to an obstacle whose movement properties change over time, and may correspond to an obstacle that has a slight movement due to factors such as wind.

[0094] Additionally, the material properties of the obstacle may correspond to various characteristics such as the current location, direction of movement, speed of movement, material, and shape of an object capable of transmitting, diffracting, or reflecting the propagation path of virtual sound. The virtual sound propagation unit (310) can perform interactions such as reflection, transmission, absorption, and diffraction according to the material properties of the obstacle and generate a sound propagation path. Here, the virtual sound propagation unit (310) can determine the sound success path as a valid path through which sound originating from the sound source location arrives at the sound sink after undergoing reflection, transmission, absorption, diffraction, etc.

[0095]

[0096] FIG. 6 is a flowchart illustrating the process of generating an M-channel sound output by inputting an N-channel original sound in a three-dimensional sound reproduction device according to the present invention.

[0097] Referring to FIG. 6, a three-dimensional sound regeneration device (130) receives attribute information regarding the location of a sound source, the location of a sound sink, and a 3D sound space (110) to perform sound propagation, and can determine at least one sound success path. Here, the attribute information of the 3D sound space (110) may correspond to the arrangement structure of static and dynamic obstacles within a virtual sound propagation space. In the process of determining the sound success path, the three-dimensional sound regeneration device (130) can determine a valid path reaching the sound sink through reflection, transmission, absorption, and diffraction according to the attribute information of the 3D sound space as the sound success path.

[0098] Next, the three-dimensional sound regeneration device (130) can perform buffering based on a specific reference time for at least one sound success path among all sound success paths through an adaptive sound path buffering unit (330). Here, the three-dimensional sound regeneration device (130) can prevent distortion of the sound success paths by arranging the sound success paths in chronological order and excluding sound success paths that arrive after a specific reference time, and by performing time synchronization of each sound success path.

[0099] Next, the 3D sound regeneration device (130) can generate sound output based on the sound source, sound sink, arrival properties of the sound success path, and material properties by performing auralization on the sound success path through the 3D virtual sound auralization unit (350). Here, the 3D sound regeneration device (130) can generate M-channel sound output by creating new virtual channels in the original sound based on the channel position of the original sound, the mixing consistency of each channel, and the panning distribution. Through this, the 3D sound regeneration device (130) can create new virtual channels in positions such as left, right, front left and right, rear left and right, and center, depending on the channel position. In addition, the 3D sound regeneration device (130) can form new virtual channels with mixing consistency by analyzing the phase relationship and spatial correlation of each channel. The three-dimensional sound reproduction device (130) can provide three-dimensional sound to the user by creating and arranging a virtual channel that provides specific instrument sounds in a virtual sound propagation space through panning distribution.

[0100]

[0101] Although the present invention has been described above with reference to preferred embodiments, those skilled in the art will understand that various modifications and changes can be made to the invention without departing from the spirit and scope of the invention as described in the following claims.

[0102]

[0103] [National R&D projects that supported this invention]

[0104] [Project ID] 1425174979

[0105] [Assignment No.] S3317348

[0106] [Ministry Name] Ministry of SMEs and Startups

[0107] [Name of Project Management (Specialized) Agency] Korea Technology Information Promotion Agency for SMEs

[0108] [Research Project Name] Startup Growth Technology Development

[0109] [Project Title] Development of Sound Rendering Semiconductor IP Supporting Dynamic Diffraction for Real-time 3D Audio in the Ultra-Realistic Metaverse

[0110] [Name of Project Performing Organization] Exarion Co., Ltd.

[0111] [Research Period] 2022.09.01 ~ 2024.08.31

[0112]

[0113] [Explanation of the symbol]

[0114] 100: 3D Sound Reproduction System

[0115] 110: 3D Sound Space 130: 3D Sound Reproduction Device

[0116] 150: Database

[0117] 210: Processor 230: Memory

[0118] 250: User I / O Section 270: Network I / O Section

[0119] 290: Communication port section

[0120] 310: Virtual Sound Propagation Unit

[0121] 330: Adaptive sound path buffering section

[0122] 350: 3D Virtual Sound Auralization Department

[0123] 370: Control unit

Claims

1. Determining at least one sound success path from the location of a sound source to the location of a sound sink in a virtual sound propagation space, wherein the sound source is a virtual sound propagation unit that generates original sound of N (where N is a natural number) channels; An adaptive sound path buffering unit that buffers the above-mentioned at least one sound success path according to a specific criterion; and A three-dimensional sound reproduction device in a virtual sound propagation space comprising a 3D virtual sound auralization unit that generates a 3D virtual sound output of M (where M is a natural number) channels in the sound sink based on the channel properties of the original sound, the arrival properties of at least one sound success path, and the material properties of the virtual sound propagation space.

2. In paragraph 1, the virtual sound propagation unit A three-dimensional sound reproduction device in a virtual sound propagation space characterized by optimizing the number of at least one sound success path based on the location of the sound sink in the virtual sound propagation space.

3. In paragraph 2, the virtual sound propagation unit A three-dimensional sound reproduction device in a virtual sound propagation space characterized by analyzing the spatial attributes of the virtual sound propagation space to determine a minimum value for the number of at least one sound success path.

4. In paragraph 1, the adaptive sound path buffering unit A three-dimensional sound reproduction device in a virtual sound propagation space characterized by selecting and buffering a sound success path that reached the sound sink from the time of generation of the original sound up to a specific reference time.

5. In paragraph 1, the 3D virtual sound auralization unit A three-dimensional sound reproduction device in a virtual sound propagation space characterized by generating a new virtual channel in the original sound based on the location of each channel constituting the channel attributes of the original sound, the mixing coherence of each channel, and the panning distribution of each channel.

6. In paragraph 1, the 3D virtual sound auralization unit A three-dimensional sound reproduction device in a virtual sound propagation space characterized by analyzing the time difference of each channel constituting the arrival attribute of at least one sound success path and deleting a specific channel in the original sound if the time difference is greater than a specific standard.

7. In paragraph 1, the 3D virtual sound auralization unit A three-dimensional sound reproduction device in a virtual sound propagation space characterized by amplifying a specific channel in the original sound by analyzing the material properties of the virtual sound propagation space that constitute the sound reflectivity of a static obstacle and the sound absorption of a dynamic obstacle placed in each of the at least one sound success paths.

Images