Spatial acoustic processing device and spatial acoustic processing method

The spatial sound processing device addresses the challenge of simulating nonlinear elements by using multiple filters to enhance realism and accuracy in sound localization, achieving a more immersive audio experience through parallel convolution and nonlinear processing.

WO2024247507A1PCT designated stage expired Publication Date: 2024-12-05JVC KENWOOD CORP

Patent Information

Application Number
PCT/JP2024/014644
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-30
Filing Date
2024-04-11
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Conventional spatial sound processing systems face challenges in reproducing a high sense of realism and accurate sound localization due to the difficulty in simulating nonlinear elements such as spatial fluctuation, which is typically addressed using a single filter coefficient, limiting the optical illusion effect.

Method used

A spatial sound processing device and method that employs a filter storage unit with multiple filters based on spatial sound transfer characteristics, convolves these filters in parallel to generate convolution signals, and performs nonlinear processing to create a fluctuation signal, enhancing the realism of sound reproduction by simulating spatial fluctuations.

Benefits of technology

The approach allows for a higher sense of realism and accurate out-of-head localization by effectively simulating spatial fluctuations, improving the overall sound reproduction experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2024014644_05122024_PF_FP_ABST
    Figure JP2024014644_05122024_PF_FP_ABST
Patent Text Reader

Abstract

This spatial acoustic processing device comprises: a filter storage unit for storing a filter set including a plurality of first filters based on one spatial acoustic transfer characteristic; convolution units (111a to 111d) for generating a plurality of first convolution signals by convolving the plurality of first filters in parallel with a first input signal of a first channel; a fluctuation signal generating unit (113) for generating a first fluctuation signal by performing nonlinear processing with respect to the plurality of convolution signals; and a filter processing unit for generating an output signal by convolving a second filter with the first fluctuation signal.
Need to check novelty before this filing date? Find Prior Art

Description

Spatial sound processing device and spatial sound processing method

[0001] The present disclosure relates to a spatial sound processing device and a spatial sound processing method.

[0002] Patent Literature 1 discloses an audio signal processing device that reproduces multi-surround audio signals as two-channel audio signals. This audio signal processing device convolves a selected head-related transfer function with the audio signal of each channel. The audio signal processing device calculates the center position of fluctuation for the audio signal of each channel and sets the fluctuation width.

[0003] International Publication No. 2013 / 183392

[0004] One sound image localization technology is out-of-head localization, which uses headphones to localize a sound image outside the listener's head. Out-of-head localization technology localizes a sound image outside the head by canceling the characteristics from the headphones to the ears and applying the characteristics of four lines from the stereo speakers to the ears.

[0005] In out-of-head localization playback, measurement signals (impulse sounds, etc.) emitted from two-channel (hereinafter referred to as ch) speakers are recorded by microphones (hereinafter referred to as mics) placed at the ears of the listener (user). Then, a processing device creates a filter based on the picked-up sound signal obtained from the impulse response. This creates a filter that corresponds to the spatial acoustic transfer characteristics from the speaker to the microphone position in the ear canal. Out-of-head localization playback can be achieved by convolving the created filter with the two-channel audio signal.

[0006] Furthermore, in order to generate a filter for canceling the characteristics from the headphones to the ear, the characteristics from the headphones to the eardrum (also called the ear canal transfer function ECTF or ear canal transfer characteristics) are measured using a microphone placed in the listener's own ear.

[0007] When sound emitted from a sound source in a real space actually reaches the ears of a listener, various factors can be nonlinear. This makes it difficult to perfectly reproduce a real environment in spatial audio processing systems such as out-of-head localization processors.

[0008] One of the nonlinear factors is spatial fluctuation (strictly speaking, everything has fluctuation, from the sound source itself to the human body). Conventional technology processes only with a unique filter coefficient at a given moment, making it difficult to simulate this fluctuation. This hinders the realization of a high sense of presence and a highly accurate illusion effect.

[0009] The present disclosure has been made in consideration of the above points, and provides a spatial audio processing device and a spatial audio processing method that are capable of reproducing audio with a high sense of realism.

[0010] The spatial acoustic processing device according to this embodiment includes a filter memory unit that stores a filter set having a plurality of first filters based on one spatial acoustic transfer characteristic, a first convolution unit that generates a plurality of first convolution signals by convolving the plurality of first filters in parallel with a first input signal of a first channel, a first fluctuation signal generation unit that generates a first fluctuation signal by performing nonlinear processing on the plurality of first convolution signals, and a filter processing unit that generates an output signal by convolving a second filter with the first fluctuation signal.

[0011] The spatial acoustic processing method according to this embodiment includes a step of reading out a filter set having a plurality of first filters based on one spatial acoustic transfer characteristic from a filter storage unit, a first convolution step unit that generates a plurality of first convolution signals by convolving the plurality of first filters in parallel with a first input signal of a first channel, a first fluctuation signal step that generates a first fluctuation signal by performing nonlinear processing on the plurality of first convolution signals, and a filter processing step that generates an output signal by convolving a second filter with the first fluctuation signal.

[0012] According to the present disclosure, it is possible to provide a spatial audio processing device and a spatial audio processing method that can reproduce sound with a high sense of realism.

[0013] Fig. 1 is a block diagram showing an out-of-head localization processing device according to the present embodiment; Fig. 2 is a schematic diagram showing the configuration of a measurement device that measures spatial acoustic transfer characteristics; Fig. 3 is a block diagram showing the configuration of a signal processing device used in the measurement processing device; Fig. 4 is a block diagram showing the configuration of a main part of a spatial acoustic processing unit; Fig. 5 is a block diagram showing the configuration of an inverse filter unit; Fig. 6 is a flowchart showing a spatial acoustic processing method according to the present embodiment;

[0014] An overview of the spatial acoustic processing according to this embodiment will be described. In this embodiment, an example of the spatial acoustic processing will be described in which out-of-head localization processing is performed using spatial acoustic transfer characteristics and ear canal transfer characteristics.

[0015] The spatial acoustic transfer characteristic is the transfer characteristic from a sound source such as a speaker to the ear canal. The ear canal transfer characteristic is the transfer characteristic from the speaker unit of headphones or earphones to the eardrum. In this embodiment, the spatial acoustic transfer characteristic is measured when headphones or earphones are not worn, and the ear canal transfer characteristic is measured when headphones or earphones are worn, and these measurement data are used to realize out-of-head localization processing. This embodiment is characterized by a microphone system for measuring the spatial acoustic transfer characteristic or the ear canal transfer characteristic.

[0016] The out-of-head localization processing according to this embodiment is executed by a user terminal such as a personal computer, a smartphone, or a tablet PC. The user terminal is an information processing device having processing means such as a processor, storage means such as a memory or a hard disk, display means such as an LCD monitor, and input means such as a touch panel, buttons, a keyboard, or a mouse. The user terminal may have a communication function for transmitting and receiving data. Furthermore, output means (output unit) having headphones or earphones is connected to the user terminal. The connection between the user terminal and the output means may be wired or wireless.

[0017] (Out-of-head localization processing device) FIG. 1 shows a block diagram of an out-of-head localization processing device 100, which is an example of a sound field reproduction device according to this embodiment. The out-of-head localization processing device 100 reproduces a sound field for a user U wearing headphones 43. To this end, the out-of-head localization processing device 100 performs sound image localization processing on left and right channel stereo input signals XL and XR. The left and right channel stereo input signals XL and XR are analog audio playback signals output from a CD (Compact Disc) player or the like, or digital audio data such as MP3 (MPEG Audio Layer-3). Note that audio playback signals or digital audio data are collectively referred to as playback signals. In other words, the left and right channel stereo input signals XL and XR are playback signals.

[0018] In this embodiment, the out-of-head localization processing device 100 performs arithmetic processing to appropriately perform sound image localization processing using a filter. The arithmetic processing unit of the out-of-head localization processing device 100 is a personal computer (PC), tablet terminal, smartphone, etc., and is equipped with a memory and a processor. The memory stores processing programs, various parameters, measurement data, etc. The processor executes the processing programs stored in the memory. Each process is performed by the processor executing the processing programs. The processor may be, for example, a CPU (Central Processing Unit), FPGA (Field-Programmable Gate Array), DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), or GPU (Graphics Processing Unit).

[0019] The out-of-head localization processing device 100 is not limited to being a single physical device, and some of the processing may be performed by different devices. For example, some of the processing may be performed by a smartphone or the like, and the remaining processing may be performed by a DSP (Digital Signal Processor) built into the headphones 43.

[0020] The out-of-head localization processing device 100 includes a spatial audio processing unit 10, an inverse filter unit 41 that stores an inverse filter Linv, an inverse filter unit 42 that stores an inverse filter Rinv, and headphones 43. Specifically, the spatial audio processing unit 10, the inverse filter unit 41, and the inverse filter unit 42 can be realized by a processor or the like.

[0021] The spatial acoustic processing unit 10 includes convolution calculation units 11-12, 21-22 that store spatial acoustic transfer characteristics Hls, Hlo, Hro, and Hrs, and adders 24 and 25. The convolution calculation units 11-12, 21-22 perform convolution processing using the spatial acoustic transfer characteristics. Stereo input signals XL and XR from an audio player or the like are input to the spatial acoustic processing unit 10. The spatial acoustic transfer characteristics are set in the spatial acoustic processing unit 10. The spatial acoustic processing unit 10 convolves the stereo input signals XL and XR of each channel with a spatial acoustic transfer characteristic filter (hereinafter also referred to as a spatial acoustic filter). The spatial acoustic transfer characteristics may be head-related transfer functions (HRTFs) measured using the subject's head or pinna, or may be head-related transfer functions of a dummy head or a third party.

[0022] A set of four spatial acoustic transfer characteristics Hls, Hlo, Hro, and Hrs is defined as a spatial acoustic transfer function. Data used for convolution in the convolution calculation units 11, 12, 21, and 22 becomes a spatial acoustic filter. A spatial acoustic filter is generated by cutting out the spatial acoustic transfer characteristics Hls, Hlo, Hro, and Hrs with a predetermined filter length.

[0023] The spatial acoustic transfer characteristics Hls, Hlo, Hro, and Hrs are acquired in advance by impulse response measurement or the like. For example, a user U wears microphones on his left and right ears, respectively. Left and right speakers arranged in front of the user U output impulse sounds for impulse response measurement. Then, measurement signals such as impulse sounds output from the speakers are collected by the microphone. The spatial acoustic transfer characteristics Hls, Hlo, Hro, and Hrs are acquired based on the signals collected by the microphone. The spatial acoustic transfer characteristic Hls between the left speaker and the left microphone, the spatial acoustic transfer characteristic Hlo between the left speaker and the right microphone, the spatial acoustic transfer characteristic Hro between the right speaker and the left microphone, and the spatial acoustic transfer characteristic Hrs between the right speaker and the right microphone are measured.

[0024] The convolution calculation unit 11 then convolves the Lch stereo input signal XL with a spatial acoustic filter corresponding to the spatial acoustic transfer characteristic Hls. The convolution calculation unit 11 outputs the convolution calculation data to the adder 24. The convolution calculation unit 21 convolves the Rch stereo input signal XR with a spatial acoustic filter corresponding to the spatial acoustic transfer characteristic Hro. The convolution calculation unit 21 outputs the convolution calculation data to the adder 24. The adder 24 adds the two convolution calculation data and outputs the result to the inverse filter unit 41.

[0025] The convolution calculation unit 12 convolves the Lch stereo input signal XL with a spatial acoustic filter corresponding to the spatial acoustic transfer characteristic Hlo. The convolution calculation unit 12 outputs the convolution calculation data to the adder 25. The convolution calculation unit 22 convolves the Rch stereo input signal XR with a spatial acoustic filter corresponding to the spatial acoustic transfer characteristic Hrs. The convolution calculation unit 22 outputs the convolution calculation data to the adder 25. The adder 25 adds the two convolution calculation data and outputs the result to the inverse filter unit 42. Details of the processing of the convolution calculation units 11 and 12 will be described later.

[0026] The inverse filters 41, 42 are set with inverse filters Linv, Rinv that cancel headphone characteristics (characteristics between the headphone playback unit and the microphone). The inverse filters Linv, Rinv are then convolved with the playback signal (convolution signal) processed by the spatial audio processing unit 10. The inverse filter 41 convolves the Lch signal from the adder 24 with the inverse filter Linv of the Lch headphone characteristics. Similarly, the inverse filter 42 convolves the Rch signal from the adder 25 with the inverse filter Rinv of the Rch headphone characteristics. When headphones 43 are worn, the inverse filters Linv, Rinv cancel the characteristics from the headphone unit to the microphone. The microphone may be placed anywhere between the entrance of the ear canal and the eardrum.

[0027] The inverse filter unit 41 outputs the processed Lch signal YL to the left unit 43L of the headphones 43. The inverse filter unit 42 outputs the processed Rch signal YR to the right unit 43R of the headphones 43. A user U wears the headphones 43. The headphones 43 output the Lch signal YL and the Rch signal YR (hereinafter, the Lch signal YL and the Rch signal YR are also collectively referred to as stereo signals or output signals) to the user U. This makes it possible to reproduce a sound image localized outside the head of the user U.

[0028] In this way, the out-head localization processing device 100 performs out-head localization processing using spatial acoustic filters corresponding to the spatial acoustic transfer characteristics Hls, Hlo, Hro, and Hrs, and inverse filters Linv and Rinv of headphone characteristics. In the following description, the spatial acoustic filters corresponding to the spatial acoustic transfer characteristics Hls, Hlo, Hro, and Hrs, and the inverse filters Linv and Rinv of headphone characteristics are collectively referred to as out-head localization processing filters. In the case of a 2-channel stereo playback signal, the out-head localization filters are composed of four spatial acoustic filters and two inverse filters. The out-head localization processing device 100 then performs convolution processing on the stereo playback signal using a total of six out-head localization filters, thereby performing out-head localization processing. It is preferable that the out-head localization filters be based on individual measurements of the user U. For example, the out-head localization filters are set based on sound signals picked up by microphones attached to the user U's ears.

[0029] In this way, the spatial acoustic filter and the headphone characteristic inverse filters Linv and Rinv are filters for audio signals. These filters are convolved with the playback signals (stereo input signals XL and XR), so that the out-of-head localization processing device 100 executes out-of-head localization processing. In this embodiment, the process of generating the spatial acoustic filter is one of the technical features. Specifically, a plurality of filters are set in each of the convolution calculation units 11, 12, 21, and 22.

[0030] For example, each of the convolution operation units 11, 12, 21, and 22 generates multiple convolution signals by simultaneously convolving an input signal with multiple filters. Furthermore, the spatial audio processing unit 10 performs nonlinear processing on the multiple convolution signals to generate a fluctuation signal. An output signal is generated by convolving the fluctuation signal with an inverse filter.

[0031] (Spatial Acoustic Transfer Characteristics Measurement Apparatus) A measurement apparatus 200 for measuring spatial acoustic transfer characteristics Hls, Hlo, Hro, and Hrs will be described with reference to Fig. 2 and Fig. 3. Fig. 2 is a diagram schematically showing a measurement configuration for measuring a subject 1. Fig. 3 is a block diagram showing the configuration of a measurement processing device 201 used in the measurement apparatus 200. Note that the subject 1 here may be the same person as user U in Fig. 1, or may be a different person.

[0032] As shown in Fig. 2, the measurement device 200 has a stereo speaker 5 and a microphone unit 2. The stereo speaker 5 is installed in a measurement environment. The measurement environment may be a room in the user U's home, an audio system sales store or showroom, or the like. The measurement environment is preferably a listening room equipped with speakers and acoustics.

[0033] In this embodiment, the measurement processing device 201 of the measurement device 200 performs arithmetic processing for appropriately generating a spatial acoustic filter. The measurement processing device 201 includes, for example, a music player such as a CD player. The measurement processing device 201 may be a personal computer (PC), a tablet terminal, a smartphone, or the like. Furthermore, the measurement processing device 201 may be a server device itself.

[0034] The stereo speakers 5 include a left speaker 5L and a right speaker 5R. For example, the left speaker 5L and the right speaker 5R are placed in front of the subject 1. The left speaker 5L and the right speaker 5R output impulse sounds or the like for performing impulse response measurement. In the following description of this embodiment, the number of speakers serving as sound sources is two (stereo speakers), but the number of sound sources used for measurement is not limited to two, and may be one or more. In other words, this embodiment can also be applied to a 1-channel monaural environment or a so-called multi-channel environment such as 5.1-channel or 7.1-channel.

[0035] The microphone unit 2 is a stereo microphone having a left microphone 2L and a right microphone 2R. The left microphone 2L is placed at the left ear 9L of the subject 1, and the right microphone 2R is placed at the right ear 9R of the subject 1. Specifically, it is preferable to place the microphones 2L and 2R at a position from the entrance of the ear canal to the eardrum of the left ear 9L and right ear 9R. The microphones 2L and 2R collect measurement signals output from the stereo speakers 5 and obtain collected signals. The microphones 2L and 2R output the collected signals to the measurement processing device 201. The subject 1 may be a person or a dummy head. In other words, in this embodiment, the subject 1 is a concept that includes not only a person but also a dummy head.

[0036] As shown in FIG. 3, the measurement processing device 201 includes a measurement signal generating unit 231 , a picked-up signal acquiring unit 232 , a filter generating unit 233 , and a filter storage unit 234 .

[0037] The measurement signal generator 231 includes a D / A converter, an amplifier, and the like, and generates a measurement signal for measuring spatial acoustic transfer characteristics. The measurement signal is, for example, an impulse signal or a TSP (Time Stretched Pulse) signal. Here, the measuring device 200 performs impulse response measurement using an impulse sound as the measurement signal.

[0038] The left microphone 2L and right microphone 2R of the microphone unit 2 each collect measurement signals and output the collected signals to the measurement processing device 201. The collected signal acquisition unit 232 acquires the collected signals collected by the left microphone 2L and right microphone 2R. The collected signal acquisition unit 232 may include an A / D converter that performs A / D conversion of the collected signals from the microphones 2L and 2R. The collected signal acquisition unit 232 may synchronously add signals obtained by multiple measurements.

[0039] As described above, the impulse response is measured by measuring the impulse sounds output from the left speaker 5L and the right speaker 5R with the microphones 2L and 2R. The measurement processing device 201 stores the picked-up signals acquired by the impulse response measurement in a memory or the like. This allows the spatial acoustic transfer characteristic Hls between the left speaker 5L and the left microphone 2L, the spatial acoustic transfer characteristic Hlo between the left speaker 5L and the right microphone 2R, the spatial acoustic transfer characteristic Hro between the right speaker 5R and the left microphone 2L, and the spatial acoustic transfer characteristic Hrs between the right speaker 5R and the right microphone 2R to be measured. That is, the spatial acoustic transfer characteristic Hls is acquired by the left microphone 2L picking up the measurement signal output from the left speaker 5L. The spatial acoustic transfer characteristic Hlo is acquired by the right microphone 2R picking up the measurement signal output from the left speaker 5L. The spatial acoustic transfer characteristic Hro is acquired by the left microphone 2L picking up the measurement signal output from the right speaker 5R. The measurement signal output from the right speaker 5R is picked up by the right microphone 2R, and the spatial acoustic transfer characteristic Hrs is acquired.

[0040] The filter generation unit 233 generates a spatial acoustic filter based on the collected sound signal. The spatial acoustic filter is generated according to the spatial acoustic transfer characteristics Hls, Hlo, Hro, and Hrs from the left and right speakers 5L and 5R to the left and right microphones 2L and 2R. For example, the filter generation unit 233 cuts out the spatial acoustic transfer characteristics Hls, Hlo, Hro, and Hrs with a predetermined filter length. The measurement processing device 201 may correct the measured spatial acoustic transfer characteristics Hls, Hlo, Hro, and Hrs.

[0041] In this way, the measurement processing device 201 generates a spatial acoustic filter to be used in the convolution calculation of the out-of-head localization processing device 100. As shown in Fig. 1, the out-of-head localization processing device 100 performs out-of-head localization processing using a spatial acoustic filter according to the spatial acoustic transfer characteristics Hls, Hlo, Hro, Hrs between the left and right speakers 5L, 5R and the left and right microphones 2L, 2R. In other words, the out-of-head localization processing is performed by convolving the spatial acoustic filter with the audio playback signal.

[0042] The measurement processing device 201 performs the same processing on the collected sound signals corresponding to the spatial acoustic transfer characteristics Hls, Hlo, Hro, and Hrs. That is, the same processing is performed on each of the four collected sound signals corresponding to the spatial acoustic transfer characteristics Hls, Hlo, Hro, and Hrs. This makes it possible to generate spatial acoustic filters corresponding to the spatial acoustic transfer characteristics Hls, Hlo, Hro, and Hrs.

[0043] Furthermore, the measurement processing device 201 generates multiple filters by performing impulse response measurements multiple times. Specifically, the measurement device 200 generates n filters corresponding to the spatial acoustic transfer characteristic Hls by performing impulse response measurements n times (n is an integer greater than or equal to 2) using the left speaker 5L and the left microphone 2L. The measurement device 200 generates n filters corresponding to the spatial acoustic transfer characteristic Hlo by performing impulse response measurements n times using the left speaker 5L and the right microphone 2R.

[0044] By performing n impulse response measurements using the right speaker 5R and the left microphone 2L, n filters corresponding to the spatial acoustic transfer characteristic Hro are generated. By performing n impulse response measurements using the right speaker 5R and the right microphone 2R, n filters corresponding to the spatial acoustic transfer characteristic Hrs are generated.

[0045] Here, the spatial acoustic transfer characteristics vary slightly depending on the measurement. For example, the measurement results of the spatial acoustic transfer characteristics vary depending on noise, the orientation of the subject's face, the state of the microphone, etc. Therefore, the filter coefficients generated differ for each measurement. In other words, by performing impulse response measurements n times on the same subject, n different spatial acoustic filters are generated for each spatial acoustic transfer characteristic. In this way, it is possible to reproduce fluctuations caused by the human body, etc., and improve the localization effect.

[0046] The filter storage unit 234 includes a memory for storing filter coefficients. The filter storage unit 234 stores a plurality of filters for each spatial acoustic transfer characteristic. For example, the filter storage unit 234 stores n filters for the spatial acoustic transfer characteristic Hls. The filter storage unit 234 stores n filters for the spatial acoustic transfer characteristic Hro. The filter storage unit 234 stores n filters for the spatial acoustic transfer characteristic Hlo. The filter storage unit 234 stores n filters for the spatial acoustic transfer characteristic Hrs. The filter storage unit 234 stores 4n filters.

[0047] Here, n filters for one characteristic are also called a filter set or a filter group. That is, a filter set includes n filters. Therefore, the filter storage unit 234 stores four filter sets.

[0048] Next, spatial audio processing using multiple filters will be described with reference to Fig. 4. Fig. 4 is a block diagram showing the configuration of the main parts of the spatial audio processing unit 10. Here, the processing for the input signal XL and the processing for the input signal XR are the same. Therefore, the processing for the input signal XL will be mainly described, and the description of the processing for the input signal XR will be omitted as appropriate.

[0049] As described above, a plurality of filters are set in the convolution operation unit 11. Here, an example will be described in which n=4 and one filter set includes four filters, but the number of filters included in a filter set is not particularly limited.

[0050] First, the convolution calculation unit 11 for the spatial acoustic transfer characteristic Hls will be described. The convolution calculation unit 11 includes four convolution units 111a to 111d and a fluctuation signal generation unit 113. Different filters are set for the four convolution units 111a to 111d. In other words, the measurement device 200 generates four filters by measuring the spatial acoustic transfer characteristic Hls four times.

[0051] The convolution units 111a to 111d convolve the input signal XL with the filters in parallel. That is, the input signal XL is convolved with four filters at the same time. The signals convolved with the filters by the convolution units 111a to 111d are called convolved signals C_HLSa to C_HLSd, respectively. The processing by the four convolution units 111a to 111d is performed in parallel.

[0052] The convolution unit 111a outputs a convolution signal C_HLSa to the fluctuation signal generation unit 113. The convolution unit 111b outputs a convolution signal C_HLSb to the fluctuation signal generation unit 113. The convolution unit 111c outputs a convolution signal C_HLSc to the fluctuation signal generation unit 113. The convolution unit 111d outputs a convolution signal C_HLSd to the fluctuation signal generation unit 113. The fluctuation signal generation unit 113 receives four convolution signals C_HLSa to C_HLSd.

[0053] As described above, the filters set in the convolution units 111a to 111d represent the spatial acoustic transfer characteristic Hls, but are generated based on different measurements. Therefore, the filters set in the convolution units 111a to 111d have different filter coefficients. Therefore, the convolution signals output from the convolution units 111a to 111d are different signals.

[0054] The fluctuation signal generation unit 113 generates the fluctuation signal F_HLS by performing nonlinear processing on the four convolution signals C_HLSa to C_HLSd. For example, the fluctuation signal generation unit 113 generates the fluctuation signal F_HLS by randomly switching between multiple convolution signals. For example, the fluctuation signal generation unit 113 uses a random number to select one convolution signal from the four convolution signals C_HLSa to C_HLSd. The fluctuation signal generation unit 113 generates the fluctuation signal F_HLS by selecting a convolution signal for each sample. By using a random number in which random changes to a value, the fluctuation signal generation unit 113 can generate a fluctuation signal that includes nonlinear fluctuations.

[0055] Furthermore, the fluctuation signal generation unit 113 may generate the fluctuation signal F_HLS by switching between multiple convolution signals C_HLSa to C_HLSd in a unique or arbitrary order. For example, the fluctuation signal generation unit 113 switches between four convolution signals at intervals of a fixed number of samples. Here, the fluctuation signal generation unit 113 may switch between the convolution signals C_HLSa, C_HLSb, C_HLSc, and C_HLSd in this order, or may switch in any order. Furthermore, when the fluctuation signal generation unit 113 switches at intervals of a predetermined number of samples, the convolution signals may be synthesized so as to be cross-faded at the switching timing. The intervals of the number of samples at which the samples are switched may be fixed or may be changed randomly.

[0056] Furthermore, the convolution signal may be multiplied by an output coefficient and integrated. For example, the output coefficients for the four convolution signals C_HLSa to C_HLSd are coefficients ka to kd, respectively. In this case, the fluctuation signal is expressed as follows (1): F_HLS=ka*C_HLSa+kb*C_HLSb+kc*C_HLSc+kd*HLSc (1)

[0057] Here, the sum of the four coefficients ka to kd is set to 1. The fluctuation signal generator 113 randomly changes the coefficients ka to kd for each sample. Alternatively, the coefficients ka to kd may be randomly changed at intervals of a fixed number of samples.

[0058] The fluctuation signal generator 113 outputs the fluctuation signal F_HLS to the adder 24. The fluctuation signal generator 113 randomly switches or combines multiple convolution signals. In this way, it becomes possible to simulate some of the nonlinearity that may occur when listening to sound from a real sound source in a real space. The calculation method of the fluctuation signal generator 113 is not limited to the above.

[0059] Next, the convolution calculation unit 12 will be described. The convolution calculation unit 12 for the spatial acoustic transfer characteristic Hlo performs the same processing as the convolution calculation unit 11. Therefore, the convolution calculation unit 12 includes four convolution units 121a to 121d and a fluctuation signal generation unit 123. Different filters are set for the four convolution units 121a to 121d. In other words, the measurement device 200 generates four filters by measuring the spatial acoustic transfer characteristic Hlo four times.

[0060] The convolution units 121a to 121d convolve the input signal XL with the filters in parallel. That is, the input signal XL is convolved with four filters at the same time. The signals convolved with the filters by the convolution units 121a to 121d are called convolved signals C_HLOa to C_HLOd, respectively. The processing by the four convolution units 121a to 121d is performed in parallel.

[0061] The fluctuation signal generation unit 123 generates the fluctuation signal F_HLO by performing nonlinear processing on the four convolution signals C_HLOa to C_HLOd. In other words, the fluctuation signal generation unit 123 generates the fluctuation signal F_HLO using the same method as the fluctuation signal generation unit 113. For example, the fluctuation signal generation unit 113 randomly switches or combines multiple convolution signals. The fluctuation signal generation unit 123 outputs the fluctuation signal F_HLO to the adder 25.

[0062] The convolution operation units 21 and 22 perform similar processing on the input signal XR. For example, the convolution operation unit 21 includes multiple convolution units 211 and a fluctuation signal generation unit 213. The multiple convolution units 221 each convolve a different filter onto the input signal XR. The fluctuation signal generation unit 213 generates a fluctuation signal F_HRO by performing nonlinear processing. The fluctuation signal generation unit 213 switches or combines multiple convolution signals, and outputs the fluctuation signal F_HRO to the adder 24. The adder 24 outputs a sum signal obtained by adding the fluctuation signal F_HLS and the fluctuation signal F_HRO to the inverse filter unit 41 of FIG. 1.

[0063] The convolution calculation unit 22 includes a plurality of convolution units 221 and a fluctuation signal generation unit 223. Each of the plurality of convolution units 221 convolves a different filter with the input signal XR. The fluctuation signal generation unit 223 generates the fluctuation signal F_HRS by performing nonlinear processing. The fluctuation signal generation unit 223 generates the fluctuation signal F_HRS by switching or combining the plurality of convolution signals. The convolution calculation unit 22 outputs the fluctuation signal F_HRS to the adder 25. The adder 25 outputs a sum signal obtained by adding the fluctuation signal F_HLO and the fluctuation signal F_HRS to the inverse filter unit 42 of FIG. 1.

[0064] In this way, for each channel, the spatial audio processing unit 10 convolves multiple filters simultaneously in parallel. The fluctuation signal generation units 113, 123, 213, and 223 generate fluctuation signals by switching or synthesizing digital signals for each sample. The fluctuation signal generation units 113, 123, 213, and 223 generate fluctuation signals by switching or synthesizing digital signals for each predetermined sample.

[0065] By doing so, it is possible to reproduce a pseudo-spatial fluctuation. Therefore, the user U can hear with a higher sense of realism, and it is possible to obtain a highly accurate out-of-head localization effect. By using random numbers, it is possible to randomly change signals and coefficients. This improves the reproducibility of spatial fluctuation.

[0066] In the above description, the spatial audio processing unit 10 generates the fluctuation signal, but the inverse filter units 41 and 42 may generate the fluctuation signal. A modification in which the inverse filter units 41 and 42 generate the fluctuation signal will be described with reference to Fig. 5. Fig. 5 is a block diagram showing the configuration of the inverse filter units 41 and 42.

[0067] The inverse filter unit 41 has multiple convolution units 411a to 411d and a fluctuation signal generation unit 413. The inverse filter unit 41 stores multiple inverse filters Linv. The convolution units 411a to 411d each store a different inverse filter Linv. The convolution units 411a to 411d then simultaneously convolve the sum signal from the adder 24 with the different inverse filters Linv in parallel.

[0068] A plurality of inverse filters Linv are generated by measuring the ear canal transfer characteristics multiple times. Note that although the number of convolution units 411a to 411d is four, it may be two or more. Of course, the number of convolution units in the inverse filter unit 41 and the number of convolution units 211, etc. may be the same or different.

[0069] The signals convolved with the inverse filter Linv by the convolution units 411a to 411d are referred to as convolution signals C_Linva to C_Linvd, respectively. The convolution unit 411a outputs the convolution signal C_Linva to the fluctuation signal generation unit 413. Similarly, the convolution units 411b to 411d output the convolution signals C_Linvb to C_Linvd to the fluctuation signal generation unit 413. The fluctuation signal generation unit 413, like the fluctuation signal generation unit 113, generates fluctuation signals from multiple convolution signals. The inverse filter unit 41 outputs the fluctuation signal generated by the fluctuation signal generation unit 413 to the left unit 43L as the Lch signal YL.

[0070] Similarly, the inverse filter unit 42 has multiple convolution units 421a to 421d and a fluctuation signal generation unit 423. The inverse filter unit 42 stores multiple inverse filters Rinv. The convolution units 421a to 421d each store a different inverse filter Rinv. The convolution units 421a to 421d then convolve the sum signal from the adder 25 with the different inverse filters Rinv simultaneously in parallel.

[0071] A plurality of inverse filters Rinv are generated by measuring the ear canal transfer characteristics multiple times. Note that although the number of convolution units 421a to 421d is four, it may be two or more. Of course, the number of convolution units in the inverse filter unit 42 may be the same as or different from the number of inverse filter units 41, convolution units 211, etc.

[0072] The signals convolved with the inverse filter Rinv by the convolution units 421a to 421d are referred to as convolution signals C_Rinva to C_Rinvd, respectively. The convolution unit 421a outputs the convolution signal C_Rinva to the fluctuation signal generation unit 423. Similarly, the convolution units 421b to 421d output the convolution signals C_Rinvb to C_Rinvd to the fluctuation signal generation unit 423. The fluctuation signal generation unit 423 generates fluctuation signals from multiple convolution signals, similar to the fluctuation signal generation units 113, 413, etc. The inverse filter unit 42 outputs the fluctuation signal generated by the fluctuation signal generation unit 423 to the right unit 43R as the output signal YR. Then, the output signals YL and YR that have been subjected to out-of-head localization processing are reproduced from the headphones 43.

[0073] In this way, the inverse filter units 41 and 42 may generate fluctuation signals. The inverse filter units 41 and 42 generate fluctuation signals by switching or combining digital signals for each sample. In this way, it is possible to reproduce pseudo-spatial fluctuations. Therefore, the user U can hear with a higher sense of realism and obtain a highly accurate out-of-head localization effect. By using random numbers, etc., it is possible to randomly change signals and coefficients, thereby improving the reproducibility of spatial fluctuations.

[0074] (Spatial Acoustic Processing Method) The spatial acoustic processing method will be described with reference to Fig. 6. Fig. 6 is a flowchart showing the spatial acoustic processing method. First, the spatial acoustic processing unit 10 reads out a filter set stored in the filter storage unit 234 (S101). As described above, a filter set includes multiple filters for one spatial acoustic transfer characteristic. In this example, the spatial acoustic processing unit 10 reads out filter sets corresponding to the spatial acoustic transfer characteristics Hls, Hlo, Hro, and Hrs. In other words, the spatial acoustic processing unit 10 reads out four filter sets.

[0075] Next, the convolution unit convolves multiple filter sets in parallel with each of the input signals XL and XR (S102). Specifically, the convolution units 111a to 111d shown in FIG. 4 convolve the input signal XL with a filter. The convolution units 121a to 121d convolve the input signal XL with a filter. The convolution unit 211 convolves the input signal XR with multiple filters. The convolution unit 221 convolves the input signal XR with multiple filters.

[0076] Then, the fluctuation signal generation units 113, 123, 213, and 223 generate fluctuation signals based on the multiple convolution signals (S103). For example, the fluctuation signal generation unit 113 performs nonlinear processing such as switching and combining multiple convolution signals to generate the fluctuation signal F_HLS. The fluctuation signal generation unit 123 performs nonlinear processing such as switching and combining multiple convolution signals to generate the fluctuation signal F_HLO. The fluctuation signal generation unit 213 performs nonlinear processing such as switching and combining multiple convolution signals to generate the fluctuation signal F_HRO. The fluctuation signal generation unit 223 performs nonlinear processing such as switching and combining multiple convolution signals to generate the fluctuation signal F_HRS.

[0077] Then, adders 24 and 25 add the fluctuation signals (S104). Adder 24 adds fluctuation signal F_HLS and fluctuation signal F_HRO and outputs the sum signal to inverse filter unit 41. Adder 25 adds fluctuation signal F_HLO and fluctuation signal F_HRS and outputs the sum signal to inverse filter unit 42.

[0078] The inverse filter units 41 and 42 convolve the sum signal with the inverse filters Linv and Rinv, respectively (S105). As a result, a playback signal that has been subjected to out-of-head localization processing is reproduced from the headphones 43. Of course, in S105 as well, the inverse filter units 41 and 42 may use fluctuation signals as shown in FIG.

[0079] In the above embodiment, the measurement device 200 generates a filter set by measuring the subject 1 multiple times, but the filter set may be generated by other methods. For example, the filter set may be generated by measuring a subject other than the user. In this case, the out-of-head localization processing device 100 may identify a subject having characteristics similar to those of the user through a predetermined matching process. Then, the out-of-head localization processing device 100 uses a filter for the subject having characteristics similar to those of the user.

[0080] For spatial acoustic transfer characteristics, a filter set may be generated for each speaker. For ear canal transfer characteristics, a filter set may be generated for each headphone or earphone. The user then selects the device to be used from the preset filter sets. This allows an appropriate filter set to be acquired. The user may also download the filter set via a network such as the Internet. This allows the user to enjoy out-of-head localization listening in a desired environment.

[0081] In the above description, the input signal is a 2-channel stereo input signal, but the input signal may be a multi-channel signal such as 5.1-channel or 7.1-channel. In this case, multiple filters may be set for each channel speaker. That is, a filter set having multiple filters is set for the spatial acoustic transfer characteristic from the speaker of each channel to the left ear. A filter set having multiple filters is set for the spatial acoustic transfer characteristic from the speaker of each channel to the right ear. The adders 24 and 25 may add three or more fluctuation signals.

[0082] Some or all of the above processes may be executed by a computer program. The above-mentioned program can be stored and provided to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). The program may also be supplied to a computer by various types of transient computer-readable media. Examples of transient computer-readable media include electric signals, optical signals, and electromagnetic waves. The transient computer-readable medium can supply the program to a computer via a wired communication path such as an electric wire or optical fiber, or via a wireless communication path.

[0083] The invention made by the inventor has been specifically described above based on an embodiment, but it goes without saying that the present invention is not limited to the above embodiment and can be modified in various ways without departing from the gist of the invention.

[0084] This application claims priority based on Japanese Patent Application No. 2023-88540, filed May 30, 2023, the disclosure of which is incorporated herein in its entirety by reference.

[0085] The present disclosure is applicable to a spatial audio processing device and a spatial audio processing method.

[0086] U User 1 Subject 2L Left microphone 2R Right microphone 5L Left speaker 5R Right speaker 9L Left ear 9R Right ear 10 Spatial acoustic processing unit 11 Convolution calculation unit 12 Convolution calculation unit 21 Convolution calculation unit 22 Convolution calculation unit 24 Adder 25 Adder 41 Inverse filter unit 42 Inverse filter unit 43 Headphones 100 Out-of-head localization processing device 111a to 111d Convolution unit 113 Fluctuation signal generation unit 121a to 121d Convolution unit 123 Fluctuation signal generation unit 200 Measurement device 201 Measurement processing device 231 Measurement signal generation unit 232 Sound pickup signal acquisition unit 233 Filter generation unit 234 Filter storage unit

Claims

1. A spatial audio processing device comprising: a filter memory unit that stores a filter set having a plurality of first filters based on one spatial audio transfer characteristic; a first convolution unit that generates a plurality of first convolution signals by convolving the plurality of first filters in parallel with a first input signal of a first channel; a first fluctuation signal generation unit that generates a first fluctuation signal by performing nonlinear processing on the plurality of first convolution signals; and a filter processing unit that generates an output signal by convolving the first fluctuation signal with a second filter.

2. The spatial audio processing device of claim 1, wherein the filter processing unit comprises: a second convolution unit that generates a plurality of second convolution signals by convolving the first fluctuating signal with a plurality of second filters in parallel; and a second fluctuating signal generation unit that generates a second fluctuating signal by performing nonlinear processing on the plurality of second convolution signals.

3. A spatial audio processing device as described in claim 2, comprising: a third convolution unit that generates a plurality of third convolution signals by convolving a second input signal of a second channel with a plurality of third filters in parallel; a third fluctuation signal generation unit that generates a third fluctuation signal by performing nonlinear processing on the plurality of third convolution signals; and an adder that outputs a sum signal obtained by adding the first fluctuation signal and the third fluctuation signal to the filter processing unit.

4. The spatial audio processing device according to claim 3, wherein the first fluctuation signal generating unit generates the first fluctuation signal by selecting one of the plurality of first convolution signals.

5. The spatial audio processing device according to claim 3, wherein the first fluctuation signal generation unit generates a first fluctuation signal by multiplying the plurality of first convolution signals by their respective output coefficients and accumulating the results.

6. A spatial sound processing device as described in any one of claims 1 to 3, further comprising a measurement processing unit that measures the spatial sound transfer characteristics of the subject multiple times, and in order to generate the multiple first filters, a first filter corresponding to each of the spatial sound transfer characteristics measured multiple times is generated.

7. A spatial acoustic processing method comprising: a step of reading out a filter set having a plurality of first filters based on one spatial acoustic transfer characteristic from a filter memory unit; a first convolution step unit that generates a plurality of first convolution signals by convolving the plurality of first filters in parallel with a first input signal of a first channel; a first fluctuation signal step that generates a first fluctuation signal by performing nonlinear processing on the plurality of first convolution signals; and a filter processing step that generates an output signal by convolving a second filter with the first fluctuation signal.

Citation Information

Patent Citations

  • Spatially Ambient-Aware Personal Audio Delivery Device

    JP2020500492A

  • Out-of-head localization filter determination system, out-of-head localization processing device, out-of-head localization filter determination device, out-of-head localization filter determination method, and program

    JP2021052315A

Cited By

  • Acoustic vibration temperature multi-modal fusion coal mill deterioration degree evaluation method based on deep learning

    CN122221217A