3D sound space equalizing device

Abstract

The present invention relates to a 3D sound space equalizing device, and the device comprises: a virtual sound propagation space generation unit for generating a virtual sound propagation space consisting of a sound source, a sound sink, and a propagation space influence medium; an impulse response calculation unit for detecting a change in components within the virtual sound propagation space and calculating an impulse response between the sound source and the sound sink; and a sound convolution unit for, when input sound data of N channels (where N is a natural number) is inputted to the sound source, performing a convolution operation on the input sound data and the impulse response so as to receive output sound data of M channels (where M is a natural number) from the sound sink.

Description

3D Sound Spatial Equalizing Device

[0001] The present invention relates to 3D sound space equalizing technology, and more specifically, to a 3D sound space equalizing device capable of calculating in real time the impulse response generated whenever a component of a virtual sound propagation space changes, and synthesizing the impulse response generated in the virtual sound propagation space with the actual response.

[0002]

[0003] Recently, interest in virtual reality technology has been rapidly increasing due to advancements in mobile technology, graphics technology, and sensory input / output technology. To support a realistic virtual reality environment, it is necessary to reproduce not only visual spatiality through virtual space but also high-quality auditory spatiality. To reproduce auditory spatiality, multi-channel audio systems or 3D sound technology using Head Related Transfer Functions (HRTF) can be used.

[0004] However, multi-channel audio used for 3D sound reproduction presents challenges, such as the need for a dedicated speaker system and space for its installation. Most 3D sound technologies based on head transfer functions utilize pre-calculated acoustic control filters or reproduce auditory spatiality within simple virtual spaces, such as rectangular shoe boxes. Since physical effects regarding the surrounding environment and the materials of complex objects in the virtual space are not reflected, these technologies have limitations in reproducing realistic sound.

[0005] To address these limitations, 3D sound technologies based on geometric or numeric methods are being introduced. Among the various geometric methods, a method that combines ray tracing technology in 3D graphics with sound processing technology is called sound tracing. Sound tracing is a type of sound rendering technique that generates sound by tracing the sound propagation paths between a listener and a sound source.

[0006] Sound tracing can generate realistic sound sources by creating various sound propagation paths, such as direct, reflection, and edge-diffraction paths, to reproduce realistic sound, but these sound propagation paths have an unstable problem when a propagation space influence medium is newly created or removed between the sound source and the listener.

[0007]

[0008] [Prior Art Literature]

[0009] [Patent Literature]

[0010] Korean Registered Patent No. 10-1828908 (2018.02.07)

[0011]

[0012] One embodiment of the present invention aims to provide a 3D sound space equalizing device capable of synthesizing the physical properties of a virtual space to a real sound source in real time by calculating the impulse response that occurs whenever a component of a virtual sound propagation space changes in real time.

[0013] One embodiment of the present invention aims to provide a 3D sound space equalizing device capable of performing 3D sound space equalizing by calculating an impulse response between a sound source and a sound sink in a virtual sound propagation space and performing sound convolution.

[0014] One embodiment of the present invention aims to provide a 3D sound spatial equalizing device capable of controlling an impulse response based on a sound success path through the reconstruction of a sound success path between a sound source and a sound sink.

[0015] One embodiment of the present invention aims to provide a 3D sound spatial equalizing device capable of generating M-channel output sound data by selecting input sound data of a major channel from N-channel input sound data and performing a convolution operation.

[0016]

[0017] Among the embodiments, the 3D sound space equalizing device comprises: a virtual sound space generation unit that generates a virtual sound space composed of a sound source, a sound sink, and a propagation space influence medium; an impulse response calculation unit that detects a change in a component in the virtual sound space and calculates an impulse response between the sound source and the sound sink; and a sound convolution unit that, when input sound data of N (where N is a natural number) channels is input to the sound source, performs a convolution operation on the input sound data and the impulse response to receive output sound data of M (where M is a natural number) channels from the sound sink.

[0018] The above virtual sound propagation space generation unit can generate the virtual sound propagation space by reflecting the location information, material information, and movement information of the propagation space influence medium.

[0019] The impulse response calculation unit can track the movement information of the propagation space influence medium to reconstruct a sound success path between the sound source and the sound sink, and control the impulse response based on the sound success path.

[0020] The above impulse response calculation unit can determine the importance of the sound success path if, through the analysis of the impulse response, the attenuation of the sound success path falls below a specific attenuation criterion due to the movement of the propagation space influence medium.

[0021] The above impulse response generating unit can generate a sound virtual path that replaces the sound success path according to the importance of the sound success path.

[0022] The sound convolution unit above can perform the convolution operation by selecting input sound data of the main channel from the input sound data of the N channels.

[0023] The sound convolution unit can generate output sound data of the M channel through the first generation of output sound data of the main channel and the second generation of output sound data of the auxiliary channel during the convolution operation process for the input sound data of the main channel.

[0024]

[0025] The disclosed technology may have the following effects. However, this does not mean that a specific embodiment must include all of the following effects or only the following effects; therefore, the scope of the rights of the disclosed technology should not be understood as being limited by this.

[0026] A 3D sound space equalizing device according to one embodiment of the present invention can play a sound source in a virtual sound space in real time by calculating the impulse response that occurs whenever a component of the virtual sound propagation space changes in real time.

[0027] A 3D sound space equalizing device according to one embodiment of the present invention can perform 3D sound space equalizing by calculating the impulse response between a sound source and a sound sink in a virtual sound propagation space and performing sound convolution.

[0028] A 3D sound spatial equalizing device according to one embodiment of the present invention can control the impulse response based on the sound success path through the reconstruction of the sound success path between the sound source and the sound sink.

[0029] A 3D sound spatial equalizing device according to one embodiment of the present invention can generate M-channel output sound data by selecting input sound data of a major channel from N-channel input sound data and performing a convolution operation.

[0030]

[0031] FIG. 1 is a drawing illustrating a 3D sound space equalizing system according to one embodiment of the present invention.

[0032] Figure 2 is a diagram illustrating the system configuration of the 3D sound space equalizing device of Figure 1.

[0033] Figure 3 is a diagram illustrating the configuration of the 3D sound space equalizing device of Figure 1.

[0034] FIG. 4 is a drawing illustrating an example of a 3D sound space equalizing device according to an embodiment of the present invention.

[0035] FIG. 5 is a flowchart illustrating the functional configuration of a 3D sound space equalizing device according to one embodiment of the present invention.

[0036] Figure 6 is a diagram illustrating the virtual sound propagation space generation unit in Figure 3.

[0037]

[0038] The description of the present invention is merely an example for structural or functional explanation, and therefore the scope of the present invention should not be interpreted as being limited by the examples described in the text. That is, since the examples are subject to various modifications and may take various forms, the scope of the present invention should be understood to include equivalents capable of realizing the technical concept. Furthermore, the objectives or effects presented in the present invention do not imply that a specific example must include all of them or only such effects; therefore, the scope of the present invention should not be understood as being limited by them.

[0039] Meanwhile, the meaning of the terms described in this application should be understood as follows.

[0040] Terms such as "first," "second," etc., are intended to distinguish one component from another, and the scope of rights shall not be limited by these terms. For example, the first component may be named the second component, and similarly, the second component may be named the first component.

[0041] When it is stated that one component is "connected" to another component, it should be understood that it may be directly connected to that other component, or that there may be other components in between. Conversely, when it is stated that one component is "directly connected" to another component, it should be understood that there are no other components in between. Meanwhile, other expressions describing the relationships between components, such as "between" and "exactly between," or "adjacent to" and "directly adjacent to," should be interpreted in the same way.

[0042] A singular expression should be understood to include a plural expression unless the context clearly indicates otherwise, and terms such as "include" or "have" are intended to specify the existence of the implemented features, numbers, steps, actions, components, parts, or combinations thereof, and should be understood not to preclude the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0043] In each step, identifiers (e.g., a, b, c, etc.) are used for convenience of explanation and do not describe the order of the steps; the steps may occur differently from the specified order unless a specific order is clearly indicated in the context. That is, the steps may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the reverse order.

[0044] The present invention may be implemented as computer-readable code on a computer-readable recording medium, and the computer-readable recording medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. Additionally, the computer-readable recording medium may be distributed across networked computer systems, so that computer-readable code can be stored and executed in a distributed manner.

[0045] Unless otherwise defined, all terms used herein have the same meaning as generally understood by those skilled in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted as having meanings consistent with the context of the relevant technology and should not be interpreted as having an ideal or overly formal meaning unless explicitly defined in this application.

[0046]

[0047] FIG. 1 is a drawing illustrating a 3D sound space equalizing system according to one embodiment of the present invention.

[0048] Referring to FIG. 1, a 3D sound space equalizing system (100) may be implemented including a 3D sound space (110), a 3D sound space equalizing device (130), and a database (150), and may perform a 3D sound space equalizing operation.

[0049] The 3D sound space (110) corresponds to an actual physical sound space that supports the implementation of a virtual sound propagation space through a 3D sound space equalizing device (130). For example, the 3D sound space (110) can be implemented as a space in a public area, such as a theater in a movie theater. As another example, the 3D sound space (110) can be implemented as a space in a private area, such as a user's headset.

[0050] The 3D sound space (110) can physically fix or variably change the location of a sound source, such as a speaker. For example, the 3D sound space (110) can physically fix the location of the sound source when occupied by multiple listeners (i.e., multiple sound sinks). For another example, the 3D sound space (110) can variably change the location of the sound source when utilized by a single listener (i.e., a single sound sink).

[0051] The 3D sound space (110) may be composed of a singular or multiple units, and if composed of multiple units, it may be composed of a first 3D sound space (e.g., 110a), a second 3D sound space (e.g., 110b), ..., an nth (n is a natural number) 3D sound space that can be independently controlled by a 3D sound space equalizing device (130).

[0052] The 3D sound space equalizing device (130) may be implemented as a computing device (e.g., a server) that performs a 3D sound space equalizing method according to one embodiment of the present invention and is implemented as a computing program. The 3D sound space equalizing device (130) can play a sound source in the virtual space in real time by calculating the impulse response that occurs whenever a component of the virtual space changes, and can allow a sound sink (e.g., a listener of 110a or 110b) to experience the three-dimensional sound effect of the sound source played in the non-existent virtual space in the actual space.

[0053] A 3D sound space equalizing device (130) can transmit sound to a sound sink (e.g., a listener of 110a or 110b) in the 3D sound space (110) by calculating an impulse response in real time, taking into account a statically fixed or dynamically moving propagation space influence medium in the 3D sound space (110) through a sound source in the 3D sound space (110) (e.g., a sound source or speaker of 110a or 110b). For example, the propagation space influence medium may correspond to an air medium, an underwater medium, or a temporary or semi-permanent obstacle in the 3D sound space (110) existing on the path from the sound source to the sound sink.

[0054] The 3D sound space equalizing device (130) can transmit and receive data with the sound source (e.g., a sound source or speaker of 110a or 110b) by connecting the sound source of the 3D sound space (110) to a wired network or a wireless network such as Bluetooth, WiFi, or LTE.

[0055] The 3D sound space equalizing device (130) can control the 3D sound space (110) independently or dependently, and can determine independent control or dependent control based on the correlation of the sound sink.

[0056] The database (150) may correspond to a storage device that stores sound source content of the 3D sound space equalizing device (130). The database (150) may provide sound source content to be delivered to the sound sink in response to a request for sound source content from the 3D sound space equalizing device (130). Although the database (150) is depicted as a device independent of the 3D sound space equalizing device (130), it is not necessarily limited thereto and can, of course, be implemented by being included in the 3D sound space equalizing device (130).

[0057]

[0058] Figure 2 is a diagram illustrating the system configuration of the 3D sound space equalizing device of Figure 1.

[0059] Referring to FIG. 2, the 3D sound space equalizing device (130) may include a processor (210), memory (230), user input / output unit (250), network input / output unit (270), and communication port unit (290).

[0060] The processor (210) can execute a 3D sound space equalizing service procedure according to an embodiment of the present invention, manage memory (230) that is read or written during this process, and schedule the synchronization time between volatile memory and non-volatile memory in memory (230). The processor (210) can control the overall operation of the 3D sound space equalizing device (130) and is electrically connected to memory (230), user input / output unit (250), and network input / output unit (270) to control the data flow between them. The processor (210) can be implemented as a CPU (Central Processing Unit) or GPU (Graphics Processing Unit) of the 3D sound space equalizing device (130).

[0061] The memory (230) may include an auxiliary storage device implemented as non-volatile memory such as an SSD (Solid State Disk) or HDD (Hard Disk Drive) and used to store all data required for the 3D sound spatial equalizing device (130), and may include a main memory device implemented as volatile memory such as RAM (Random Access Memory). Additionally, the memory (230) may store a set of instructions that execute a 3D sound spatial equalizing method according to an embodiment of the present invention by being executed by an electrically connected processor (210).

[0062] The user input / output unit (250) includes an environment for receiving user input and an environment for outputting specific information to the user, and may include an input device including an adapter such as a touch pad, touch screen, virtual keyboard, or pointing device, and an output device including an adapter such as a monitor or touch screen. In one embodiment, the user input / output unit (250) may correspond to a computing device connected via remote access, and in such case, the 3D sound spatial equalizing device (130) may be performed as an independent server.

[0063] The network input / output unit (270) provides a communication environment for connecting to a specific terminal through a network and may include an adapter for communication such as a LAN (Local Area Network), MAN (Metropolitan Area Network), WAN (Wide Area Network), and VAN (Value Added Network). Additionally, the network input / output unit (270) may be implemented to provide short-range communication functions such as WiFi and Bluetooth, or wireless communication functions of 4G or higher, for wireless transmission of learning data.

[0064] The communication port (290) is a hardware interface for connecting to external hardware, for example, the external hardware may include a printer, a mouse, and USB hardware. The communication port (290) can detect the connection of specific USB hardware and perform the role of a 3D sound spatial equalizing device (130).

[0065]

[0066] Figure 3 is a diagram illustrating the configuration of the 3D sound space equalizing device of Figure 1.

[0067] Referring to FIG. 3, the 3D sound space equalizing device (130) may include a virtual sound propagation space generating unit (310), an impulse response calculating unit (330), a sound convolution unit (350), and a control unit (370).

[0068] The 3D sound spatial equalizing device (130) does not necessarily have to include all of the above functional components simultaneously; depending on each embodiment, some of the components may be omitted, or some or all of the components may be selectively included. Additionally, the 3D sound spatial equalizing device (130) may be implemented as an independent module that selectively includes some of the above components, and the 3D sound spatial equalizing method according to an embodiment of the present invention may be performed through the interaction between each module. The operation of each component will be described in detail below.

[0069]

[0070] The virtual sound propagation space generation unit (310) can generate a virtual sound propagation space composed of a sound source, a sound sink, and a propagation space influence medium on the 3D sound space (110). The 3D sound space (110) can be implemented as a space in a plaza area with a physical sound space or as a space in a private area with a logical sound space.

[0071] A sound source may correspond to a virtual point where sound is generated in a 3D sound space (110). For example, a sound source may correspond to a virtual gunshot point where a gunshot is generated in a virtual game implemented in the 3D sound space (110), or a virtual speech point where voice is generated in a virtual meeting implemented in the 3D sound space (110). The attribute information of the sound source may include various characteristics such as the location, intensity, directionality, and frequency band of the virtual sound, and each may change during the sound propagation process depending on static and dynamic propagation space influence media.

[0072] A sound sink may correspond to a virtual point that consumes (i.e., listens to) sound implemented in a 3D sound space (110). For example, a sound sink may correspond to a virtual listening point that listens to gunshots in a virtual game implemented in the 3D sound space (110), or a virtual speaking point that listens to voices in a virtual conference implemented in the 3D sound space (110). The attribute information of the sound sink may include various characteristics such as the listening location, intensity, directionality, and frequency band of the virtual sound, and each sound sink attribute information may change during the sound propagation process depending on static and dynamic propagation space influence media.

[0073] A propagation space influence medium may correspond to a virtual object that distorts the sound propagation path implemented in the 3D sound space (110). For example, a propagation space influence medium may correspond to a statically fixed cover or a dynamically moving tank that distorts the propagation path of a gunshot in a virtual game implemented in the 3D sound space (110), or a statically fixed wall or a dynamically moving other person that distorts the propagation path of a voice in a virtual conference implemented in the 3D sound space (110). The attribute information of the propagation space influence medium may include various characteristics such as the current location, direction of movement, speed of movement, material, and shape of an object capable of transmitting, diffracting, reflecting, and reflecting the propagation path of virtual sound.

[0074] The virtual sound propagation space generation unit (310) can generate a virtual sound propagation space capable of simulating the interaction of sound propagation within a virtual space based on attribute information of a sound source, a sound sink, and a propagation space influence medium. For example, the virtual sound propagation space generation unit (310) can generate a virtual sound propagation space capable of learning the path of sound that is reflected, absorbed, refracted, and scattered according to the location of the sound source, the sound sink, and the propagation space influence medium. Here, the virtual sound propagation space generation unit (310) can perform sound propagation simulation according to the propagation space influence medium included in the path during the process in which sound generated from a sound source is transmitted to a sound sink. For example, the virtual sound propagation space generation unit (310) can control the sound transmitted to the sound sink according to the degree to which the sound is reverberated by a propagation space influence medium including a wall.

[0075] In one embodiment, the virtual sound propagation space generation unit (310) can perform sound propagation simulation according to the degree of sound absorption and sound shielding caused by the propagation space influence medium during the process in which sound generated from a sound source is transmitted to a sound sink. For example, the virtual sound propagation space generation unit (310) can simulate the degree to which sound is absorbed or shielded by the propagation space influence medium including a sound-absorbing material and control the sound transmitted to the sound sink. In addition, the virtual sound propagation space generation unit (310) can control the sound transmitted to the sound sink by performing sound propagation simulation regarding sound distortion including refraction and scattering caused by the propagation space influence medium.

[0076] In one embodiment, the virtual sound propagation space generation unit (310) can generate a virtual sound propagation space by reflecting location information, material information, and movement information of the propagation space influence medium. Here, the virtual sound propagation space generation unit (310) can adjust the degree to which sound is reflected, refracted, absorbed, and shielded according to the location of the propagation space influence medium. For example, the virtual sound propagation space generation unit (310) can calculate a path for sound to bypass the propagation space influence medium according to the location of the propagation space influence medium and simulate the delay and volume reduction of sound that occur accordingly.

[0077] Additionally, the virtual sound propagation space generation unit (310) can determine the degree of reflection, absorption, and transmission during the process of sound interacting with the propagation space influence medium according to the material information of the propagation space influence medium and control the sound. For example, the virtual sound propagation space generation unit (310) can generate an echo when sound is reflected according to the material information of the propagation space influence medium, and is not necessarily limited to this, but can control the sound in a direction such as weakening the reflected sound or reducing the reverberation of the sound when sound is absorbed.

[0078] Additionally, the virtual sound propagation space generation unit (310) can adjust the sound propagation path in real time according to the movement information of the propagation space influence medium within a specific space. Here, the virtual sound propagation space generation unit (310) can dynamically adjust the reflection, absorption, and shielding of sound when the sound propagation path changes according to the movement path and movement speed of the propagation space influence medium. For example, the virtual sound propagation space generation unit (310) can convert the location information of the propagation space influence medium into coordinates in a three-dimensional space and calculate coordinate information corresponding to the location of the propagation space influence medium through a coordinate transformation operation on a three-dimensional coordinate system with a reference point in the three-dimensional space as the origin. Here, the virtual sound propagation space generation unit (310) can determine the sound propagation path from the sound source to the sound sink based on the coordinate information of the propagation space influence medium and adjust it according to the movement of the propagation space influence medium coordinate information. In addition, the virtual sound propagation space generation unit (310) can adjust the sound based on the acoustic propagation path that changes according to the movement of the sound source or sound sink, as well as the propagation space influence medium.

[0079] The impulse response calculation unit (330) can calculate the impulse response between the sound source and the sound sink by detecting changes in components in the virtual sound propagation space. Here, the impulse response may correspond to a response measured over time along the sound propagation path, for example, it may correspond to a response including reflected sound, refracted sound, and absorption that occurred from the time the sound is generated at the sound source to the time it reaches the sound sink. The impulse response calculation unit (330) can identify acoustic characteristics based on the attribute information of the sound source, the sound sink, and the propagation space influence medium and calculate the impulse response. Here, the impulse response calculation unit (330) can track the entire acoustic propagation path between the sound source and the sound sink based on the virtual sound propagation space and calculate interactions such as reflection, absorption, scattering, and refraction occurring in each path.

[0080] In one embodiment, the impulse response calculation unit (330) can reconstruct a sound success path between a sound source and a sound sink by tracking the movement information of a propagation space influence medium and control the impulse response based on the sound success path. Here, the sound success path may correspond to the path of the sound that actually reaches the sound sink among the entire acoustic propagation path. The impulse response calculation unit (330) can reconstruct the sound success path in the entire acoustic propagation path by considering the process of sound being reflected, absorbed, scattered, and refracted by the propagation space influence medium. Through this, the impulse response calculation unit (330) can control the characteristics of the sound reaching the sound sink, including the arrival time, intensity, and timbre of the sound.

[0081] In one embodiment, the impulse response calculation unit (330) can determine the importance of the sound success path through the analysis of the impulse response if the attenuation of the sound success path due to the movement of the propagation space influence medium falls below a specific attenuation standard. Here, the impulse response calculation unit (330) can determine the importance of the sound success path if the acoustic level resulting from the movement of the propagation space influence medium falls below a certain standard. For example, the impulse response calculation unit (330) can measure the acoustic level of the sound success path by classifying it into reflected sound and direct sound and determine the importance. Here, the impulse response calculation unit (330) can measure the acoustic level of the sound success path corresponding to the reflected sound and classify the importance of the sound success path as low if it falls below a specific attenuation standard.

[0082] In one embodiment, the impulse response calculation unit (330) can generate a sound virtual path that replaces the sound success path according to the importance of the sound success path. Here, the sound virtual path may correspond to an alternative sound transmission path generated when the sound success path fails to perform a desired level of sound transmission due to external factors such as propagation space influence media. The impulse response calculation unit (330) can generate a sound virtual path that compensates for attenuation for the sound success path of high importance. Additionally, the impulse response calculation unit (330) is not necessarily limited to this and can compensate for the sound success path of high importance to maintain acoustic directionality. Through this, the impulse response calculation unit (330) can optimize and transmit the sound transmitted to the user sync based on the sound success path of high importance.

[0083] The sound convolution unit (350) can receive M (M is a natural number) channel output sound data from the sound sink by performing a convolution operation on the input sound data and the impulse response when N (N is a natural number) channel input sound data is input to the sound source. Here, the input sound data may correspond to the original sound signal and, for example, may be classified into mono sound, stereo sound, and surround sound depending on the number of channels. The sound convolution unit (350) can process the input sound data by reflecting the acoustic characteristics in the virtual environment by performing a convolution operation on the input sound data and the impulse response. For example, the sound convolution unit (350) can perform a convolution operation by matching the impulse response with each sample of the input sound data. Here, the sample may correspond to the result of converting an analog acoustic signal into a digital acoustic signal, for example, may correspond to the sound intensity at a specific moment.

[0084] In one embodiment, the sound convolution unit (350) can perform a convolution operation by selecting input sound data of a major channel from N-channel input sound data. Here, the sound convolution unit (350) can determine the major channel by distinguishing a channel containing major acoustic elements from the N-channel input sound data. For example, the sound convolution unit (350) can distinguish the input sound data according to the type of sound such as dialogue, background sound, etc., and determine the channel that processes input sound data corresponding to at least one of the sound types as the major channel. The sound convolution unit (350) can reflect acoustic characteristics in a virtual environment by performing a convolution operation on the input sound data and impulse response of the major channel.

[0085] In one embodiment, the sound convolution unit (350) can generate an M-channel output sound by generating a primary output sound data of the main channel and a secondary output sound data of the auxiliary channel during the convolution operation process for the input sound data of the main channel. Here, the sound convolution unit (350) can generate an output sound suitable for a specific virtual environment by determining at least one main channel and performing a convolution operation with the input sound data of the main channel and the impulse response.

[0086] Next, the sound convolution unit (350) can determine at least one channel as an auxiliary channel and perform a convolution operation between the input sound data of the auxiliary channel and the impulse response. Here, the auxiliary channel may correspond to a channel intended to complement the main channel or provide additional acoustic elements, for example, it may correspond to a surround channel. The sound convolution unit (350) can perform convolution between the input sound data of the auxiliary channel and the impulse response by considering the position, direction, reverberation, and reverberation effects of the output sound data of the main channel generated in the first step. Through this, the sound convolution unit (350) can generate output sound data of the auxiliary channel that is balanced with the output sound data of the main channel.

[0087] In one embodiment, the sound convolution unit (350) can finally generate an M-channel output sound by performing EQ (Equalization) adjustment based on the first and second generated output sound data. Here, the sound convolution unit (350) can perform frequency response flattening for the first and second generated output sound data through EQ adjustment. Through this, the sound convolution unit (350) can optimize the first and second generated output sound data for a specific virtual space, and is not necessarily limited to this, but can adjust the pitch, volume, tempo, and spatial position of the sound for the final output sound by performing EQ adjustment through a separate terminal.

[0088] The control unit (370) controls the overall operation of the 3D sound space equalizing device (130) and can manage the control flow or data flow between the virtual sound propagation space generation unit (310), the impulse response calculation unit (330), and the sound convolution unit (350).

[0089]

[0090] FIG. 4 is a drawing illustrating an example of a 3D sound space equalizing device according to an embodiment of the present invention.

[0091] Referring to FIG. 4, the 3D sound space equalizing device (130) can create a virtual sound propagation space on the 3D sound space (110) based on the virtual sound propagation space generating unit (310) and perform environment settings for the virtual sound propagation space. For example, the 3D sound space equalizing device (130) can set the virtual sound propagation space environment by forming a propagation space influence medium including a current location, direction of movement, speed of movement, material, and shape on the virtual sound propagation space. In addition, the 3D sound space equalizing device (130) is not necessarily limited to this and can adjust the intensity, directionality, and frequency band of the sound source and sound sink by performing environment settings for the attribute information of the sound source and sound sink.

[0092] Additionally, the 3D sound spatial equalizing device (130) can receive N-channel input sound data based on the audio input unit (410). Here, the audio input unit (410) may correspond to a microphone, a recording device, etc., and may correspond to a device that receives an audio signal from an external source, not necessarily limited to these. The 3D sound spatial equalizing device (130) can receive N-channel input sound data based on the audio input unit (410) and store it in the database (150).

[0093] The 3D sound space equalizing device (130) can detect in real time the impulse response that occurs when a component of the 3D sound space (110) changes based on the impulse response calculation unit (330). The 3D sound space equalizing device (130) can play sound in real time in a virtual sound propagation space by deriving a sound success path based on acoustic characteristics according to changes in the attribute information of the sound source, sound sink, and propagation space influence medium. Here, the 3D sound space equalizing device (130) can optimize the sound delivered to the user sink by deriving importance for each sound success path and creating a virtual sound path that replaces the sound success path according to importance.

[0094] The 3D sound spatial equalizing device (130) can convolve the impulse response from the impulse response based on the sound convolution unit (350) with the input sound data received from the audio input unit (410). Here, the 3D sound spatial equalizing device (130) can distinguish channels containing major acoustic elements from N-channel input sound data to determine them as major channels, and can reflect acoustic characteristics in a virtual environment by performing a convolution operation on the input sound data and impulse response of the major channels. The 3D sound spatial equalizing device (130) can finally output M-channel output sound data based on the audio output unit (430). Here, the 3D sound spatial equalizing device (130) can provide three-dimensional sound to the user by adjusting the frequency and spatial characteristics of the output sound.

[0095]

[0096] FIG. 5 is a flowchart illustrating the functional configuration of a 3D sound space equalizing device according to one embodiment of the present invention.

[0097] Referring to FIG. 5, the 3D sound space equalizing device (130) can generate a virtual sound propagation space composed of a sound source, a sound sink, and a propagation space influence medium based on a virtual sound propagation space generating unit (310) (step S510). The 3D sound space equalizing device (130) can calculate an impulse response between the sound source and the sound sink by detecting a change in a component in the virtual sound propagation space through an impulse response calculation unit (330) (step S530).

[0098] When N-channel input sound data is input to a sound source through the sound convolution unit (350), the 3D sound spatial equalizing device (130) can receive M-channel output sound data from the sound sink by performing a convolution operation on the input sound data and the impulse response (step S550).

[0099]

[0100] Figure 6 is a diagram illustrating the virtual sound propagation space generation unit in Figure 3.

[0101] In FIG. 6, the virtual sound propagation space generation unit (310) can generate a sound propagation path according to the properties of the propagation space influence medium. Here, the virtual sound propagation space generation unit (310) can generate a sound propagation path by distinguishing it according to the movement properties of the propagation space influence medium and the material properties of the propagation space influence medium. For example, the virtual sound propagation space generation unit (310) can classify the propagation space influence medium into a static propagation space influence medium, a dynamic propagation space influence medium, and a quasi-static propagation space influence medium according to the movement properties of the propagation space influence medium. Here, the quasi-static propagation space influence medium may correspond to a propagation space influence medium whose movement properties change over time, and may correspond to a propagation space influence medium that has a fine movement due to factors such as wind.

[0102] Additionally, the material properties of the propagation space influence medium may correspond to various characteristics such as the current location, direction of movement, speed of movement, material, and shape of an object capable of transmitting, diffracting, reflecting-diffracting, and reflecting the propagation path of virtual sound. The virtual sound propagation space generation unit (310) can generate an acoustic propagation path by performing interactions such as reflection, reflection-diffracting, transmission, absorption, and diffracting according to the material properties of the propagation space influence medium. Here, the virtual sound propagation space generation unit (310) can determine the sound success path as a valid path through which sound originating from the sound source location arrives at the sound sink after passing through reflection, transmission, absorption, diffracting, and reflection-diffracting.

[0103]

[0104] Although the present invention has been described above with reference to preferred embodiments, those skilled in the art will understand that various modifications and changes can be made to the invention without departing from the spirit and scope of the invention as described in the following claims.

[0105]

[0106] [National R&D projects that supported this invention]

[0107] [Project ID] 1425174979

[0108] [Assignment No.] S3317348

[0109] [Ministry Name] Ministry of SMEs and Startups

[0110] [Name of Project Management (Specialized) Agency] Korea Technology Information Promotion Agency for SMEs

[0111] [Research Project Name] Startup Growth Technology Development

[0112] [Project Title] Development of Sound Rendering Semiconductor IP Supporting Dynamic Diffraction for Real-time 3D Audio in the Ultra-Realistic Metaverse

[0113] [Name of Project Performing Organization] Exarion Co., Ltd.

[0114] [Research Period] 2022.09.01 ~ 2024.08.31

[0115]

[0116] [Explanation of the symbol]

[0117] 100: 3D Sound Spatial Equalizing System

[0118] 110: 3D Sound Space 130: 3D Sound Space Equalizing Device

[0119] 150: Database

[0120] 210: Processor 230: Memory

[0121] 250: User I / O Section 270: Network I / O Section

[0122] 290: Communication port section

[0123] 310: Virtual Sound Propagation Space Generator

[0124] 330: Impulse response output unit

[0125] 350: Sound Convolution Section

[0126] 370: Control unit

[0127] 410: Audio input 430: Audio output

Claims

1. A virtual sound propagation space generating unit that generates a virtual sound propagation space composed of a sound source, a sound sink, and a propagation space influence medium; An impulse response calculation unit that detects a change in a component in the virtual sound propagation space and calculates an impulse response between the sound source and the sound sink; and A 3D sound spatial equalizing device comprising a sound convolution unit that, when input sound data of N (where N is a natural number) channels is input to the sound source, performs a convolution operation on the input sound data and the impulse response to receive output sound data of M (where M is a natural number) channels from the sound sink.

2. In paragraph 1, the virtual sound propagation space generating unit A 3D sound space equalizing device characterized by generating the virtual sound propagation space by reflecting the location information, material information, and movement information of the propagation space influence medium.

3. In paragraph 2, the impulse response calculating unit A 3D sound space equalizing device characterized by tracking movement information of the above-mentioned propagation space influence medium to reconstruct a sound success path between the sound source and the sound sink, and controlling the impulse response based on the sound success path.

4. In paragraph 3, the impulse response calculating unit A 3D sound space equalizing device characterized by determining the importance of the sound success path when the attenuation of the sound success path falls below a specific attenuation criterion due to the movement of the propagation space influence medium through the analysis of the impulse response.

5. In paragraph 4, the impulse response calculating unit A 3D sound spatial equalizing device characterized by generating a sound virtual path that replaces the sound success path according to the importance of the sound success path.

6. In paragraph 1, the sound convolution part A 3D sound spatial equalizing device characterized by selecting input sound data of a major channel from the input sound data of the above N channels and performing the above convolution operation.

7. In paragraph 1, the sound convolution part A 3D sound spatial equalizing device characterized by generating output sound data of the M channel through the primary generation of output sound data of the main channel and the secondary generation of output sound data of the auxiliary channel during a convolution operation process for input sound data of the main channel.

Images