Stereophonic sound playback device for optimizing characteristics of virtual sound propagation space

Abstract

The present invention relates to a stereophonic sound playback device for optimizing characteristics of a sound propagation space, the device comprising: a quasi-static sound propagation space model generation unit for generating a sound propagation space model by modeling a quasi-static sound propagation space having location information between a sound sink and a sound source that generates an original sound of N (N is a natural number) channels; a sound impulse response propagation unit for determining, on the basis of the sound propagation space model, the number of sound success paths by analyzing an impulse response (IR) to the original sound of the N channels; a sound path adaptation unit for buffering the determined at least one sound success path according to a specific criterion; and a virtual stereophonic sound output unit for generating a sound output of M channels (M is a natural number) on the basis of a channel attribute of the original sound, an arrival attribute of the at least one sound success path, and a material attribute of the sound propagation space.

Description

3D sound reproduction device optimizing the characteristics of the virtual sound propagation space

[0001] The present invention relates to a stereoscopic sound reproduction technology, and more specifically, to a stereoscopic sound reproduction device that optimizes the characteristics of a virtual sound propagation space to generate a virtual stereoscopic sound output for an original sound.

[0002]

[0003] Recently, interest in virtual reality technology has been rapidly increasing due to advancements in mobile technology, graphics technology, and sensory input / output technology. To support a realistic virtual reality environment, it is necessary to reproduce not only visual spatiality through virtual space but also high-quality auditory spatiality. To reproduce auditory spatiality, multi-channel audio systems or 3D sound technology using Head Related Transfer Functions (HRTF) can be used.

[0004] However, multi-channel audio used for 3D sound reproduction presents challenges, such as the need for a dedicated speaker system and space for its installation. Most 3D sound technologies based on head transfer functions utilize pre-calculated acoustic control filters or reproduce auditory spatiality within simple virtual spaces, such as rectangular shoe boxes. Since physical effects regarding the surrounding environment and the materials of complex objects in the virtual space are not reflected, these technologies have limitations in reproducing realistic sound.

[0005] To address these limitations, 3D sound technologies based on geometric or numeric methods are being introduced. Among the various geometric methods, a method that combines ray tracing technology in 3D graphics with sound processing technology is called sound tracing. Sound tracing is a type of sound rendering technique that generates sound by tracing the sound propagation paths between a listener and a sound source.

[0006] Sound tracing can generate realistic sound sources by creating various sound propagation paths, such as direct, reflection, and edge-diffraction paths, to reproduce realistic sound, but these sound propagation paths have an unstable problem when obstacles are newly created or removed between the sound source and the listener.

[0007]

[0008] [Prior Art Literature]

[0009] [Patent Literature]

[0010] Korean Registered Patent No. 10-1828908 (2018.02.07)

[0011]

[0012] One embodiment of the present invention aims to provide a stereoscopic sound reproduction device with optimized characteristics of a sound propagation space capable of generating a virtual stereoscopic sound output for an original sound by optimizing the characteristics of the virtual sound propagation space.

[0013] One embodiment of the present invention aims to provide a three-dimensional sound reproduction device that optimizes the characteristics of a sound propagation space and can generate a sound propagation space model by reflecting the location information, material information, and mobility of quasi-static obstacles in the quasi-static sound propagation space.

[0014] One embodiment of the present invention aims to provide a three-dimensional sound reproduction device that optimizes the characteristics of a sound propagation space to generate a virtual sound path according to the importance of the sound success path due to attenuation of the sound success path caused by quasi-static obstacles.

[0015] One embodiment of the present invention aims to provide a stereoscopic sound reproduction device that optimizes the characteristics of a sound propagation space capable of generating a virtual stereoscopic sound output based on the channel properties of the original sound, the arrival properties of the sound success path, and the material properties of the sound propagation space.

[0016]

[0017] Among the embodiments, a sound propagation space model is generated by modeling a quasi-static sound propagation space having positional information between a sound source and a sound sink, and the sound source comprises a quasi-static sound propagation space model generation unit that generates an original sound of N (where N is a natural number) channels, a sound impulse response propagation unit that determines the number of sound success paths by analyzing the impulse response (IR) for the original sound of N channels based on the sound propagation space model, a sound path adaptation unit that buffers at least one determined sound success path according to a specific criterion, and a virtual stereo sound output unit that generates a virtual stereo sound output of M (where M is a natural number) channels from the sound sink based on the channel properties of the original sound, the arrival properties of the at least one sound success path, and the material properties of the sound propagation space.

[0018] The above-mentioned quasi-static sound propagation space model generation unit can generate the sound propagation space model by reflecting the location information, material information, and mobility of quasi-static obstacles in the above-mentioned quasi-static sound propagation space.

[0019] The sound impulse response propagation unit can reconstruct the sound success path by reflecting the mobility of the quasi-dynamic obstacle and calculate the impulse response based on the material information of the quasi-static obstacles.

[0020] The sound impulse response propagation unit can determine the importance of a specific sound success path if, through the analysis of the impulse response, the attenuation of the specific sound success path due to the quasi-static obstacle falls below a specific attenuation criterion.

[0021] The sound impulse response propagation unit described above can generate a sound virtual path that replaces the specific sound success path according to the importance of the specific sound success path.

[0022] The sound path adaptation unit can select and buffer a sound success path that reached the sound sink from the time of occurrence of the original sound up to a specific reference time.

[0023] The above virtual stereoscopic sound output unit may (1) create a new virtual channel in the original sound based on the location of each channel constituting the channel properties of the original sound, the mixing coherence of each channel, and the panning distribution of each channel, or (2) analyze the time difference of each channel constituting the arrival properties of at least one sound success path and delete a specific channel in the original sound if the time difference is greater than a certain standard, or (3) amplify a specific channel in the original sound by analyzing the material properties of the sound propagation space constituting the sound reflectance of static obstacles and the sound absorption of dynamic obstacles placed in each of the at least one sound success paths.

[0024]

[0025] The disclosed technology may have the following effects. However, this does not mean that a specific embodiment must include all of the following effects or only the following effects; therefore, the scope of the rights of the disclosed technology should not be understood as being limited by this.

[0026] A stereoscopic sound reproduction device that optimizes the characteristics of a sound propagation space according to one embodiment of the present invention can generate a virtual stereoscopic sound output for an original sound by optimizing the characteristics of a virtual sound propagation space.

[0027] A stereoscopic sound reproduction device that optimizes the characteristics of a sound propagation space according to one embodiment of the present invention can generate a sound propagation space model by reflecting the location information, material information, and mobility of quasi-static obstacles in a quasi-static sound propagation space.

[0028] A three-dimensional sound reproduction device that optimizes the characteristics of a sound propagation space according to one embodiment of the present invention can generate a virtual sound path based on the importance of the sound success path due to attenuation of the sound success path caused by quasi-static obstacles.

[0029] A stereoscopic sound reproduction device with optimized characteristics of a sound propagation space according to one embodiment of the present invention can generate a virtual stereoscopic sound output based on the channel properties of the original sound, the arrival properties of the sound success path, and the material properties of the sound propagation space.

[0030]

[0031] FIG. 1 is a drawing illustrating a three-dimensional sound reproduction system according to the present invention.

[0032] Figure 2 is a diagram illustrating the system configuration of the stereoscopic sound reproduction device of Figure 1.

[0033] Figure 3 is a diagram illustrating the configuration of the stereoscopic sound reproduction device of Figure 1.

[0034] FIG. 4 is a flowchart illustrating the functional configuration of a three-dimensional sound reproduction device according to the present invention.

[0035] Figure 5 is a diagram illustrating the sound impulse response propagation section in Figure 3.

[0036] FIG. 6 is a diagram illustrating the process of converting N-channel input sound data into M-channel output sound data in a stereoscopic sound reproduction device according to the present invention.

[0037]

[0038] The description of the present invention is merely an example for structural or functional explanation, and therefore the scope of the present invention should not be interpreted as being limited by the examples described in the text. That is, since the examples are subject to various modifications and may take various forms, the scope of the present invention should be understood to include equivalents capable of realizing the technical concept. Furthermore, the objectives or effects presented in the present invention do not imply that a specific example must include all of them or only such effects; therefore, the scope of the present invention should not be understood as being limited by them.

[0039] Meanwhile, the meaning of the terms described in this application should be understood as follows.

[0040] Terms such as "first," "second," etc., are intended to distinguish one component from another, and the scope of rights shall not be limited by these terms. For example, the first component may be named the second component, and similarly, the second component may be named the first component.

[0041] When it is stated that one component is "connected" to another component, it should be understood that it may be directly connected to that other component, or that there may be other components in between. Conversely, when it is stated that one component is "directly connected" to another component, it should be understood that there are no other components in between. Meanwhile, other expressions describing the relationships between components, such as "between" and "exactly between," or "adjacent to" and "directly adjacent to," should be interpreted in the same way.

[0042] A singular expression should be understood to include a plural expression unless the context clearly indicates otherwise, and terms such as "include" or "have" are intended to specify the existence of the implemented features, numbers, steps, actions, components, parts, or combinations thereof, and should be understood not to preclude the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0043] In each step, identifiers (e.g., a, b, c, etc.) are used for convenience of explanation and do not describe the order of the steps; the steps may occur differently from the specified order unless a specific order is clearly indicated in the context. That is, the steps may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the reverse order.

[0044] The present invention may be implemented as computer-readable code on a computer-readable recording medium, and the computer-readable recording medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. Additionally, the computer-readable recording medium may be distributed across networked computer systems, so that computer-readable code can be stored and executed in a distributed manner.

[0045] Unless otherwise defined, all terms used herein have the same meaning as generally understood by those skilled in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted as having meanings consistent with the context of the relevant technology and should not be interpreted as having an ideal or overly formal meaning unless explicitly defined in this application.

[0046]

[0047] FIG. 1 is a drawing illustrating a three-dimensional sound reproduction system according to the present invention.

[0048] Referring to FIG. 1, a stereoscopic sound reproduction system (100) may be implemented including a 3D sound space (110), a stereoscopic sound reproduction device (130), and a database (150), and may perform a three-dimensional sound reproduction operation.

[0049] The 3D sound space (110) corresponds to an actual physical sound space that supports the implementation of a virtual sound propagation space through a stereoscopic sound reproduction device (130). For example, the 3D sound space (110) can be implemented as a space in a plaza area, such as a movie theater. As another example, the 3D sound space (110) can be implemented as a space in a private area, such as a user's headset.

[0050] The 3D sound space (110) can physically fix or variably change the location of a sound source, such as a speaker. For example, the 3D sound space (110) can physically fix the location of the sound source when occupied by a large number of people (i.e., a large number of sound sinks). For another example, the 3D sound space (110) can variably change the location of the sound source when utilized by a single user (i.e., a single sound sink).

[0051] The 3D sound space (110) may be composed of a singular or multiple units, and if composed of multiple units, it may be composed of a first 3D sound space (e.g., 110a), a second 3D sound space (e.g., 110b), ..., an nth (n is a natural number) 3D sound space that can be independently controlled by a stereoscopic sound reproduction device (130).

[0052] The three-dimensional sound reproduction device (130) may be implemented as a computing device (e.g., a server) that performs a three-dimensional sound reproduction method according to one embodiment of the present invention and is implemented as a computing program. The three-dimensional sound reproduction device (130) can calculate a sound success path in real time corresponding to a path for transmitting sound generated from the location of a sound source to the location of a sound sink, and can play a sound source in a virtual space in real time, and can allow a sound sink (e.g., a listener of 110a or 110b) to experience the three-dimensional sound effect of the sound source played in a non-existent virtual space in a real space.

[0053] The stereoscopic sound reproduction device (130) can transmit sound to a sound sink (e.g., a listener of 110a or 110b) in the 3D sound space (110) by calculating an impulse response in real time through a sound source (e.g., a sound source or speaker of 110a or 110b) in the 3D sound space (110) and taking into account a statically fixed or dynamically moving obstacle in the 3D sound space (110).

[0054] The stereoscopic sound reproduction device (130) can transmit and receive data with the sound source (e.g., a sound source or speaker of 110a or 110b) by connecting to a sound source of the 3D sound space (110) via a wired network or a wireless network such as Bluetooth, WiFi, or LTE.

[0055] The stereoscopic sound reproduction device (130) can control the 3D sound space (110) independently or dependently, and can determine independent or dependent control based on the correlation of sound sinks. Here, the correlation of sound sinks may correspond to acoustic connectivity based on the location of the sound sinks. For example, the stereoscopic sound reproduction device (130) can control at least one sound sink located in the 3D sound space (110), such as a movie theater, dependently to deliver the same sound to each sound sink. Additionally, the stereoscopic sound reproduction device (130) can control the 3D sound space (110) independently in the case of an environment where different sound effects need to be provided.

[0056] The database (150) may correspond to a storage device that stores sound source content of the stereoscopic sound reproduction device (130). The database (150) may provide sound source content to be delivered to the sound sink in response to a request for sound source content from the stereoscopic sound reproduction device (130). Although the database (150) is depicted as a device independent of the stereoscopic sound reproduction device (130), it is not necessarily limited thereto and can, of course, be implemented by being included in the stereoscopic sound reproduction device (130).

[0057]

[0058] Figure 2 is a diagram illustrating the system configuration of the stereoscopic sound reproduction device of Figure 1.

[0059] Referring to FIG. 2, the stereoscopic sound reproduction device (130) may include a processor (210), memory (230), user input / output unit (250), network input / output unit (270), and communication port unit (290).

[0060] The processor (210) can execute a stereoscopic sound regeneration service procedure according to an embodiment of the present invention, manage memory (230) that is read or written during this process, and schedule the synchronization time between volatile memory and non-volatile memory in memory (230). The processor (210) can control the overall operation of the stereoscopic sound regeneration device (130) and is electrically connected to the memory (230), user input / output unit (250), and network input / output unit (270) to control the data flow between them. The processor (210) can be implemented as a CPU (Central Processing Unit) or GPU (Graphics Processing Unit) of the stereoscopic sound regeneration device (130).

[0061] The memory (230) may include an auxiliary storage device implemented as non-volatile memory such as an SSD (Solid State Disk) or HDD (Hard Disk Drive) and used to store all data required for the stereoscopic sound regeneration device (130), and may include a main memory device implemented as volatile memory such as RAM (Random Access Memory). Additionally, the memory (230) may store a set of instructions that execute a stereoscopic sound regeneration method according to an embodiment of the present invention by being executed by an electrically connected processor (210).

[0062] The user input / output unit (250) includes an environment for receiving user input and an environment for outputting specific information to the user, and may include an input device including an adapter such as a touch pad, touch screen, virtual keyboard, or pointing device, and an output device including an adapter such as a monitor or touch screen. In one embodiment, the user input / output unit (250) may correspond to a computing device connected via remote access, and in such case, the stereoscopic sound reproduction device (130) may be performed as an independent server.

[0063] The network input / output unit (270) provides a communication environment for connecting to a specific terminal through a network and may include an adapter for communication such as a LAN (Local Area Network), MAN (Metropolitan Area Network), WAN (Wide Area Network), and VAN (Value Added Network). Additionally, the network input / output unit (270) may be implemented to provide short-range communication functions such as WiFi and Bluetooth, or wireless communication functions of 4G or higher, for wireless transmission of learning data.

[0064] The communication port section (290) is a hardware interface for connecting to external hardware, for example, the external hardware may include a printer, a mouse, and USB hardware. The communication port section (290) can detect the connection of specific USB hardware and perform the role of a stereoscopic sound reproduction device (130).

[0065]

[0066] Figure 3 is a diagram illustrating the configuration of the stereoscopic sound reproduction device of Figure 1.

[0067] Referring to FIG. 3, the stereoscopic sound reproduction device (130) may include a quasi-static sound propagation space model generation unit (310), a sound impulse response propagation unit (330), a sound path adaptation unit (350), a virtual stereoscopic sound output unit (370), and a control unit (390).

[0068] The three-dimensional sound reproduction device (130) does not necessarily have to include all of the above functional components simultaneously, and depending on each embodiment, some of the components may be omitted, or some or all of the components may be selectively included. Additionally, the three-dimensional sound reproduction device (130) may be implemented as an independent module that selectively includes some of the components, and the three-dimensional sound reproduction method according to the present invention may be performed through the interlocking of each module. The operation of each component will be described in detail below.

[0069]

[0070] The quasi-static sound propagation space model generation unit (310) can generate a sound propagation space model by modeling a quasi-static sound propagation space having location information between a sound source and a sound sink that generate N-channel original sound. Here, the sound source may correspond to a point where sound is generated, and may include, for example, a point where a gunshot is generated in a game or a point where a specific person speaks in a virtual meeting. In addition, the sound source may include various characteristics such as the location of sound generation, sound intensity, directionality, and frequency band. The sound sink may correspond to a listener who is the subject listening to the sound generated from the sound source.

[0071] A quasi-static sound propagation space may correspond to a sound propagation space in which characteristics change over time. The quasi-static sound propagation space model generation unit (310) can track the location information of the sound source and sound sink in real time within the quasi-static sound propagation space that includes the sound source and sound sink. Here, the quasi-static sound propagation space model generation unit (310) can calculate coordinate information corresponding to the location of the sound source and sound sink by converting the location information of the sound source and sound sink into coordinates in a three-dimensional space.

[0072] Additionally, the quasi-static sound propagation space model generation unit (310) can generate a sound success path from a sound source to a sound sink based on coordinate information in a three-dimensional space. Here, the sound success path may correspond to a valid path through which sound originating from the sound source location arrives at the sound sink after undergoing reflection, transmission, absorption, and diffraction. In one embodiment, the quasi-static sound propagation space model generation unit (310) can generate a sound propagation space model that simulates the sound success path by modeling the quasi-static sound propagation space based on location information of the sound source and the sound sink.

[0073] For example, the semi-static sound propagation space model generation unit (310) can simulate a sound propagation path by considering the attribute information of obstacles within the semi-static sound propagation space. Here, the attribute information of obstacles may correspond to factors that affect the propagation of sound in a specific space, and may be classified, for example, by material, location, and shape. Additionally, the sound propagation path may correspond to a path through which sound generated from a sound source passes through obstacles or reflective surfaces and is transmitted to a sound sink, and may be classified, for example, into a direct path, a reflected path, a refraction path, and an obstructed path. The semi-static sound propagation space model generation unit (310) can generate a sound propagation space model capable of learning sound propagation paths that are reflected, absorbed, refraction, and scattered according to the attribute information of obstacles, and can simulate the sound propagation path.

[0074] In one embodiment, the quasi-static sound propagation space model generation unit (310) can generate a sound propagation space model by reflecting location information and material information of static obstacles in the quasi-static sound propagation space and reflecting the mobility of dynamic obstacles. Here, the quasi-static sound propagation space model generation unit (310) can adjust the degree of sound reflection, refraction, absorption, and shielding according to the location of static and dynamic obstacles. For example, the quasi-static sound propagation space model generation unit (310) can calculate the path by which sound bypasses obstacles according to the location of static and dynamic obstacles and simulate the sound delay and volume reduction that occur accordingly. Additionally, the sound propagation space model generation unit (310) can generate an echo when sound reflection occurs according to the material information of static obstacles, and can adjust the sound in a direction that weakens the reflected sound or reduces the reverberation of the sound when sound is absorbed by static obstacles.

[0075] Additionally, the quasi-static sound propagation space model generation unit (310) can generate a sound propagation space model that reflects the mobility of dynamic obstacles and calculates the degree of change in sound propagation according to the movement of dynamic obstacles. For example, the quasi-static sound propagation space model generation unit (310) can analyze the degree of change in sound propagation for movement including the velocity, direction, and acceleration of dynamic obstacles. For example, the quasi-static sound propagation space model generation unit (310) can adjust the acoustics according to the degree of sound shielding when the sound propagation path is blocked according to the movement of dynamic obstacles.

[0076] Additionally, the quasi-static sound propagation space model generation unit (310) can generate a sound propagation space model that applies a Doppler effect according to the movement of a dynamic obstacle. For example, the quasi-static sound propagation space model generation unit (310) can apply a Doppler effect by changing the frequency of the sound when the dynamic obstacle moves toward a sound source or a sound sink. Here, the quasi-static sound propagation space model generation unit (310) can generate a sound propagation space model that applies the intensity of the Doppler effect differentially according to the speed of the dynamic obstacle.

[0077] The sound impulse response propagation unit (330) can determine the number of sound success paths by analyzing the impulse response (IR) for the original sound of N channels based on the sound propagation space model. Here, the impulse response may correspond to a response measured over time according to the sound success path, for example, it may correspond to a response including reflected sound, refracted sound, and absorption that occurred from the time sound is generated at the sound source to the time it reaches the sound sink. The sound impulse response propagation unit (330) can identify acoustic characteristics based on the attribute information of the sound source, sound sink, and obstacle based on the sound propagation space model and calculate the impulse response. Here, the sound impulse response propagation unit (330) can track the sound success path between the sound source and the sound sink based on the sound propagation space model and calculate interactions such as reflection, absorption, scattering, and refraction occurring in each sound success path.

[0078] In one embodiment, the sound impulse response propagation unit (330) can analyze the impulse response to the original sound and determine the sound success path based on the sound arrival time and the degree of sound attenuation. Here, the sound impulse response propagation unit (330) can verify the validity of the sound arrival time of a specific sound propagation path based on the distance between the sound source and the sound sink and determine whether it is a sound success path. The sound impulse response propagation unit (330) can determine a sound propagation path corresponding to within a specific reference time as a sound success path. That is, the sound impulse response propagation unit (330) can determine a sound propagation path including a sound arrival time within the range that the sound sink can recognize as a sound success path.

[0079] Additionally, the sound impulse response propagation unit (330) can analyze the impulse response, perform a validity verification of the sound attenuation degree of a specific sound propagation path according to the attenuation degree of the original sound, and determine whether it is a sound successful path. For example, the sound impulse response propagation unit (330) can exclude a sound propagation path if the sound is attenuated below a specific standard due to reflection or shielding, and is not necessarily limited to this, but can exclude a sound propagation path if frequency characteristics such as sound distortion are changed.

[0080] In one embodiment, the sound impulse response propagation unit (330) can determine sound success paths in the entire sound propagation path according to the sound arrival time and the degree of sound attenuation, and determine the number of sound success paths according to the position of the sound sink. Here, the sound impulse response propagation unit (330) can optimize the sound success paths by selecting sound success paths that have optimal reflection paths according to the position of the sound sink. For example, if the sound propagation space corresponds to a space having multiple reflection paths, such as a concert hall, the sound impulse response propagation unit (330) can perform optimization of the sound success paths by preferentially selecting sound success paths that have sound directionality and sound quality above a certain level.

[0081] In one embodiment, the sound impulse response propagation unit (330) can reconstruct the sound success path by reflecting the mobility of dynamic obstacles and calculate the impulse response based on material information of static and dynamic obstacles. The sound impulse response propagation unit (330) can reconstruct the sound success path in the entire sound propagation path by considering the process of sound being reflected, absorbed, scattered, and refracted by obstacles. Through this, the sound impulse response propagation unit (330) can control the characteristics of the sound reaching the sound sink, including the arrival time, intensity, and timbre of the sound.

[0082] In one embodiment, the sound impulse response propagation unit (330) can determine the importance of a specific sound success path by analyzing the impulse response if the attenuation of the specific sound success path due to a dynamic obstacle falls below a specific attenuation criterion. Here, the sound impulse response propagation unit (330) can determine the importance of the sound success path if the acoustic level due to the movement of the obstacle falls below a certain criterion. For example, the sound impulse response propagation unit (330) can measure the acoustic level of the sound success path by distinguishing it into reflected sound and direct sound and determine the importance. Here, the sound impulse response propagation unit (330) can measure the acoustic level of the sound success path corresponding to the reflected sound and classify the importance of the sound success path as low if it falls below a specific attenuation criterion.

[0083] In one embodiment, the sound impulse response propagation unit (330) can generate a sound virtual path that replaces a specific sound success path according to the importance of the specific sound success path. Here, the sound virtual path may correspond to an alternative sound transmission path generated when the sound success path fails to perform a desired level of sound transmission due to external factors such as obstacles. The sound impulse response propagation unit (330) can generate a sound virtual path that compensates for attenuation for a sound success path of high importance. Additionally, the sound impulse response propagation unit (330) is not necessarily limited to this and can compensate to maintain acoustic directionality for a sound success path of high importance. Through this, the sound impulse response propagation unit (330) can optimize and transmit the sound transmitted to the user sync based on the sound success path of high importance.

[0084] The sound path adaptation unit (350) can buffer at least one determined sound success path according to a specific criterion. Here, buffering may correspond to storing sound data to adjust for delays, reflections, and acoustic distortions occurring in the sound success path. Additionally, the specific criterion may correspond to a reference value for applying buffering, for example, time delay, sound quality, and phase. By performing buffering based on the specific criterion for at least one sound success path, the sound path adaptation unit (330) can temporarily store the path and process it appropriately at the required time. For example, the sound path adaptation unit (330) can perform time synchronization of the sound success path by performing buffering on the sound success path where a time delay occurs when a time delay occurs between the sound success path corresponding to the direct path and the sound success path corresponding to the indirect path.

[0085] In one embodiment, the sound path adaptation unit (350) can select and buffer sound success paths that have reached the sound sink from the time of occurrence of the original sound up to a specific reference time. Here, the sound path adaptation unit (330) can store the sound success paths that have reached the sound sink from the time of occurrence of the original sound in a chronological order in the database (150). The sound path adaptation unit (330) can exclude sound success paths that reach the sound sink after a specific reference time and perform buffering on sound success paths that reach the sound sink up to a specific reference time. Through this, the sound path adaptation unit (330) can prevent distortion of the sound success paths by excluding sound success paths after a specific reference time when multiple sound success paths occur.

[0086] The virtual stereoscopic sound output unit (370) can generate M-channel sound output based on the channel properties of the original sound, the arrival properties of at least one sound success path, and the material properties of the sound propagation space. Here, auralization may correspond to a technology that simulates the characteristics of sound through a computer and reproduces them audibly, for example, a technology that calculates how sound will be heard within a specific space. Additionally, the arrival properties may correspond to the distance, propagation time, sound pressure level, reflection, and absorption of the sound, and are not necessarily limited thereto, but may correspond to physical properties occurring in the path from the sound source to the sound sink. The material properties may correspond to the materials of the sound source, sound sink, and obstacles within the sound propagation space. The virtual stereoscopic sound output unit (350) can generate mono, stereo, and surround channel sound output based on the arrival properties of the sound success path and the material properties of the sound propagation space.

[0087] In one embodiment, the virtual stereoscopic sound output unit (370) can create a new virtual channel in the original sound based on the position of each channel constituting the channel attributes of the original sound, the mixing coherence of each channel, and the panning distribution of each channel. Here, the channel attributes may include the number of channels, channel positions, channel mixing, sampling rate, and bit depth, but are not necessarily limited thereto. The virtual stereoscopic sound output unit (350) can create a new virtual channel according to the position of the channel. Here, the virtual stereoscopic sound output unit (350) can create a new virtual channel including positions such as left, right, front left and right, rear left and right, and center.

[0088] Additionally, the virtual stereoscopic sound output unit (350) can form a new virtual channel in the original sound based on the mixing consistency of each channel. Here, the mixing consistency may correspond to the degree of harmony of the phase relationship and interaction between each channel. The virtual stereoscopic sound output unit (350) can analyze the phase relationship of each channel and determine whether to cancel out or amplify sound in a specific frequency band. That is, the virtual stereoscopic sound output unit (350) can create a new virtual channel that can play the original sound simultaneously without a time difference based on the mixing consistency of each channel. In one embodiment, the virtual stereoscopic sound output unit (350) can analyze the spatial correlation of each channel sound and create a new virtual channel having mixing consistency. That is, the virtual stereoscopic sound output unit (350) can create a new virtual channel that provides sound generated in the same environment by adjusting the spatial correlation of each channel sound.

[0089] In one embodiment, the virtual stereoscopic sound output unit (350) can create a new virtual channel in the original sound based on a panning distribution. Here, panning may correspond to a process of determining the position of the channel sound within the sound propagation space. The virtual stereoscopic sound output unit (350) can create a spatial image of the sound propagation space by creating a new virtual channel in the original sound through the panning distribution. For example, the virtual stereoscopic sound output unit (350) can provide a stereoscopic sound to the user by creating a virtual channel that provides a specific instrument sound through the panning distribution and placing it in the sound propagation space.

[0090] In one embodiment, the virtual stereoscopic sound output unit (370) analyzes the time difference of each channel constituting the arrival attribute of at least one sound success path, and if the time difference exceeds a specific standard, it can delete a specific channel in the original sound. Here, the virtual stereoscopic sound output unit (350) can determine the average value between the channel having the minimum time and the channel having the maximum time based on a specific standard during the process of analyzing the time difference of each channel. The virtual stereoscopic sound output unit (350) analyzes the time difference of each channel based on the average value of the channels and can remove the corresponding channel if the time difference exceeds a specific standard. Through this, the virtual stereoscopic sound output unit (350) can minimize acoustic distortion caused by unnecessary delayed sound or reflected sound.

[0091] In one embodiment, the virtual stereoscopic sound output unit (370) can amplify a specific channel in the original sound by analyzing the material properties of the sound propagation space, which constitute the sound reflectance of a static obstacle and the sound absorption of a dynamic obstacle placed in each of at least one sound success path. Here, sound reflectance may correspond to the ratio of sound reflected as the sound collides with the obstacle, and may be expressed as a value between 0 and 1, for example. Additionally, sound absorption may correspond to the ratio of sound absorbed by the obstacle as the sound collides with the obstacle, and may correspond to a complementary concept to sound reflectance. The virtual stereoscopic sound output unit (350) can amplify a specific channel in the original sound by imparting sound reflectance and sound absorption according to the materials of the static obstacle and the dynamic obstacle. For example, the virtual stereoscopic sound output unit (350) can amplify a specific channel and form an echo and reverb space by forming a sound propagation space with high reflectance according to the materials of the static obstacle and the dynamic obstacle.

[0092] The control unit (390) controls the overall operation of the stereoscopic sound reproduction device (130) and can manage the control flow or data flow between the quasi-static sound propagation space model generation unit (310), the sound impulse response propagation unit (330), the sound path adaptation unit (350), and the virtual stereoscopic sound output unit (370).

[0093]

[0094] FIG. 4 is a flowchart illustrating the functional configuration of a three-dimensional sound reproduction device according to the present invention.

[0095] Referring to FIG. 4, the stereoscopic sound regeneration device (130) can generate a sound propagation space model by modeling a quasi-static sound propagation space having position information between a sound source and a sound sink that generate N-channel original sound based on a quasi-static sound propagation space model generation unit (310) (step S410). The stereoscopic sound regeneration device (130) can determine the number of sound success paths by analyzing the impulse response (IR) for the N-channel original sound based on the sound propagation space model through a sound impulse response propagation unit (330) (step S430).

[0096] The stereoscopic sound regeneration device (130) can buffer at least one sound success path determined through the sound path adaptation unit (350) according to a specific standard (step S450). The stereoscopic sound regeneration device (130) can generate an M-channel sound output based on the channel properties of the original sound, the arrival properties of at least one sound success path, and the material properties of the sound propagation space through the virtual stereoscopic sound output unit (370) (step S470).

[0097]

[0098] Figure 5 is a diagram illustrating the sound impulse response propagation section in Figure 3.

[0099] In FIG. 5, the sound impulse response propagation unit (330) can generate a sound propagation path according to the properties of the obstacle. Here, the sound impulse response propagation unit (330) can generate a sound propagation path by distinguishing it according to the movement properties of the obstacle and the material properties of the obstacle. For example, the sound impulse response propagation unit (330) can classify the obstacle into static obstacles, dynamic obstacles, and quasi-static obstacles according to the movement properties of the obstacle. Here, a quasi-static obstacle may correspond to an obstacle whose movement properties change over time, and may correspond to an obstacle that has a slight movement due to factors such as wind.

[0100] Additionally, the material properties of the obstacle may correspond to various characteristics such as the current location, direction of movement, speed of movement, material, and shape of an object capable of transmitting, diffracting, or reflecting the propagation path of virtual sound. The sound impulse response propagation unit (330) can perform interactions such as reflection, transmission, absorption, and diffraction according to the material properties of the obstacle and generate an acoustic propagation path. Here, the sound impulse response propagation unit (330) can determine the valid path through which sound originating from the sound source location arrives at the sound sink after undergoing reflection, transmission, absorption, diffraction, etc., as the sound success path.

[0101]

[0102] FIG. 7 is a diagram illustrating the process of converting N-channel input sound data into M-channel output sound data in a stereoscopic sound reproduction device according to the present invention.

[0103] Referring to FIG. 7, a stereoscopic sound regeneration device (130) receives attribute information regarding the location of a sound source, the location of a sound sink, and a 3D sound space (110) to perform sound propagation, and can determine at least one sound success path. Here, the attribute information of the 3D sound space (110) may correspond to the arrangement structure of static and dynamic obstacles within a virtual sound propagation space. In the process of determining the sound success path, the stereoscopic sound regeneration device (130) can determine a valid path reaching the sound sink through reflection, transmission, absorption, and diffraction according to the attribute information of the 3D sound space as the sound success path.

[0104] Next, the stereoscopic sound regeneration device (130) can perform buffering based on a specific reference time for at least one sound success path among all sound success paths through the sound path adaptation unit (330). Here, the stereoscopic sound regeneration device (130) can prevent distortion of the sound success paths by arranging the sound success paths in chronological order and excluding sound success paths that arrive after a specific reference time, and by performing time synchronization of each sound success path.

[0105] Next, the three-dimensional sound regeneration device (130) can generate sound output based on the sound source, sound sink, arrival properties of the sound success path, and material properties by performing auralization on the sound success path through the virtual three-dimensional sound output unit (350). Here, the three-dimensional sound regeneration device (130) can generate M-channel sound output by creating a new virtual channel in the original sound based on the channel position of the original sound, the mixing consistency of each channel, and the panning distribution. Through this, the three-dimensional sound regeneration device (130) can create new virtual channels in positions such as left, right, front left and right, rear left and right, and center depending on the channel position. In addition, the three-dimensional sound regeneration device (130) can form a new virtual channel with mixing consistency by analyzing the phase relationship and spatial correlation of each channel. The three-dimensional sound regeneration device (130) can provide three-dimensional sound to the user by creating and placing a virtual channel that provides a specific instrument sound in the virtual sound propagation space through the panning distribution.

[0106]

[0107] Although the present invention has been described above with reference to preferred embodiments, those skilled in the art will understand that various modifications and changes can be made to the invention without departing from the spirit and scope of the invention as described in the following claims.

[0108]

[0109] [National R&D projects that supported this invention]

[0110] [Project ID] 1425174979

[0111] [Assignment No.] S3317348

[0112] [Ministry Name] Ministry of SMEs and Startups

[0113] [Name of Project Management (Specialized) Agency] Korea Technology Information Promotion Agency for SMEs

[0114] [Research Project Name] Startup Growth Technology Development

[0115] [Project Title] Dynamic Diffraction for Real-time 3D Audio in the Ultra-Realistic Metaverse

[0116] Development of sound rendering semiconductor IP that supports

[0117] [Name of Project Performing Organization] Exarion Co., Ltd.

[0118] [Research Period] 2022.09.01 ~ 2024.08.31

[0119]

[0120] [Explanation of the symbol]

[0121] 100: 3D Sound Reproduction System

[0122] 110: 3D Sound Space 130: Stereoscopic Sound Reproducer

[0123] 150: Database

[0124] 210: Processor 230: Memory

[0125] 250: User I / O Section 270: Network I / O Section

[0126] 290: Communication port section

[0127] 310: Quasi-static Sound Propagation Space Model Generation Unit

[0128] 330: Sound Impulse Response Propagation Section

[0129] 350: Sound Path Adaptation

[0130] 370: Virtual 3D Sound Output Unit

[0131] 390: Control unit

Claims

A quasi-static sound propagation space model generation unit that generates a sound propagation space model by modeling a quasi-static sound propagation space having positional information between a sound source and a sound sink, wherein the sound source generates an original sound of N (where N is a natural number) channels; A sound impulse response propagation unit that determines the number of sound success paths by analyzing the impulse response (IR) for the original sound of the N channel based on the sound propagation space model above; A sound path adaptation unit that buffers at least one sound success path determined above according to a specific criterion; and A stereoscopic sound reproduction device that optimizes the characteristics of a sound propagation space, comprising a virtual stereoscopic sound output unit that generates a virtual stereoscopic sound output of M (where M is a natural number) channels at the sound sink based on the channel properties of the original sound, the arrival properties of at least one sound success path, and the material properties of the sound propagation space. In claim 1, the above-mentioned quasi-static sound propagation space model generating unit A three-dimensional sound reproduction device that optimizes the characteristics of a sound propagation space, characterized by generating a sound propagation space model by reflecting the location information, material information, and mobility of quasi-static obstacles in the quasi-static sound propagation space. In paragraph 2, the sound impulse response propagation unit A three-dimensional sound reproduction device that optimizes the characteristics of a sound propagation space, characterized by reconstructing the sound success path by reflecting the mobility of the quasi-static obstacles and calculating the impulse response based on the material information of the quasi-static obstacles. In paragraph 3, the sound impulse response propagation unit A three-dimensional sound reproduction device that optimizes the characteristics of a sound propagation space, characterized by determining the importance of a specific sound success path when the attenuation of a specific sound success path due to a quasi-static obstacle falls below a specific attenuation criterion through the analysis of the impulse response. In paragraph 4, the sound impulse response propagation unit A three-dimensional sound reproduction device that optimizes the characteristics of a sound propagation space, characterized by generating a sound virtual path that replaces the specific sound success path according to the importance of the specific sound success path. In claim 1, the sound path adaptation unit A three-dimensional sound reproduction device that optimizes the characteristics of a sound propagation space, characterized by selecting and buffering a sound success path that reached the sound sink from the time of generation of the original sound up to a specific reference time. In paragraph 1, the virtual stereoscopic sound output unit (1) Create a new virtual channel in the original sound based on the location of each channel constituting the channel properties of the original sound, the mixing coherence of each channel, and the panning distribution of each channel, or (2) Analyze the time difference of each channel constituting the arrival properties of at least one sound success path and delete a specific channel in the original sound if the time difference is greater than a certain standard, or (3) Analyze the material properties of the sound propagation space constituting the sound reflectance of static obstacles and the sound absorption of dynamic obstacles placed in each of the at least one sound success path and amplify a specific channel in the original sound, characterized by optimizing the characteristics of the sound propagation space.

Images