Sound generation device, sound reproduction device, sound generation method, and sound signal processing program

Abstract

A sound generation device (2) is provided with a direction acquisition unit (10) and a sound image translation unit (20). A direction acquisition unit (10) acquires a sound source direction of a sound source (S). On the basis of the sound source direction acquired by the direction acquisition unit (10), a sound image translation unit (20) performs sound image translation for representing the sound source (S) by performing sound image translation of sound from a specific representative direction by time shifting and gain adjustment of the sound source (S).

Description

Technical Field

[0001] The present disclosure relates to a sound generating device, a sound reproducing device, a sound generating method, and a sound signal processing program for producing a sound signal to be reproduced, in particular, using headphones or the like. Background Art

[0002] In the past, there were VR headsets and HMDs (Head Mounted Displays) that can reproduce movies, VR (Virtual Reality), AR (Augmented Reality), etc. In order to feel a wider sound field, such VR headsets and HMDs use a head transfer function (HRTF) that takes into account the direction from the listener to the sound source to perform head positioning.

[0003] As an example of a sound processing device for calculating such an HRTF, Patent Document 1 describes a device including the following structure: a sensor that outputs a detection signal corresponding to the posture of the listener's head; a sensor signal processing unit that calculates the direction in which the listener's head is facing by performing an operation based on the detection signal, and outputs direction information indicating the direction; a sensor output correction unit that corrects the direction information output from the sensor signal processing unit based on average information obtained by averaging the direction information; a head transfer function correction unit that corrects a pre-calculated head transfer function according to the corrected direction information; and sound image localization processing that performs sound image localization processing on the sound signal of the reproduced object according to the corrected head transfer function.

[0004] Here, conventionally, when stereo sound using HRTF is reproduced by headphones or the like, a head impulse response (hereinafter referred to as "HRIR") expressing the head transfer function on a time axis is often used in calculations of actual sound signals.

[0005] Prior art literature

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Publication No. 2021-5822 Summary of the invention

[0008] Problem that the invention aims to solve

[0009] In the conventional sound processing device described in Patent Document 1, HRIR convolution is performed for each sound source. Therefore, when the number of sound sources is large, HRIR convolution needs to be performed for each sound source, which increases the calculation load.

[0010] The present disclosure is made in view of such circumstances, and aims to solve the above-mentioned problems.

[0011] Means used to solve problems

[0012] A sound generating device according to one embodiment of the present disclosure includes: a direction acquisition unit that acquires a sound source direction of a sound source; and a sound image shifting unit that performs sound image shifting of a sound from a specific representative direction by time shifting and gain adjustment of the sound source based on the sound source direction acquired by the direction acquisition unit, thereby expressing the sound source.

[0013] Effects of the Invention

[0014] According to the present disclosure, a sound generating device can be provided, which synthesizes the sound source by panning the sound image of a specific representative direction based on the sound source direction, thereby equivalently generating the HRIR of the sound source direction through the HRIR of the representative direction, and can generate stereo sound of the HRIR with reduced calculation load. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a control structure diagram of the sound generating device according to the first embodiment.

[0016] Figure 2 It means by Figure 1 A conceptual diagram showing the concept of HRIR synthesis performed by the panning unit shown.

[0017] Figure 3 This is a flowchart of the sound reproduction process according to the first embodiment.

[0018] Figure 4 This is a diagram for explaining the synthesis of HRIR in the sound reproduction process according to the first embodiment.

[0019] Figure 5 This is a control structure diagram of another sound generating device according to the first embodiment.

[0020] Figure 6 It is a graph showing the comparison results of the SNR of the person's HRTF (4 directions_diagonal, right ear) in Example 1.

[0021] Figure 7 It is a graph showing the comparison results of the SNR of the person's HRTF (4 directions_diagonal, left ear) in Example 1.

[0022] Figure 8 It is a graph showing the comparison results of the SNR of the person's HRTF (4 directions_vertical and horizontal, right ear) in Example 1.

[0023] Fig. 9It is a graph showing the comparison results of the SNR of the person's HRTF (4 directions_vertical and horizontal, left ear) in Example 1.

[0024] Fig.10 This is a graph showing the comparison results of the SNR of the user's HRTF (6 directions, right ear) in Example 1.

[0025] Fig.11 This is a graph showing the comparison results of the SNR of the user's HRTF (6 directions, left ear) in Example 1.

[0026] Fig.12 This is a graph showing the results of the positioning experiment (true value) based on subjective evaluation in Example 1.

[0027] Fig.13 It is a graph showing the results of the positioning experiment (4 directions_oblique direction) based on subjective evaluation of Example 1.

[0028] Fig.14 It is a graph showing the results of the positioning experiment (4 directions - vertical and horizontal) based on subjective evaluation of Example 1.

[0029] Fig.15 This is a graph showing the results of the positioning experiment (six directions) based on subjective evaluation in Example 1.

[0030] Fig.16 : is a graph showing the results of subjective quality evaluation using the MUSHRA method in Example 1.

[0031] Fig.17 It is a graph showing the comparison results of SNR of FABIAN (4 directions_oblique direction) of Example 1.

[0032] Fig.18 It is a graph showing the comparison results of SNR of FABIAN (4 directions_vertical and horizontal) of Example 1.

[0033] Fig.19 This is a graph showing the comparison results of SNR of FABIAN (6 directions) of Example 1.

[0034] Fig. 20 This is a graph showing the comparison results of SNR of FABIAN (3 types, right ear) of Example 1.

[0035] Fig.21 This is a graph showing the comparison results of SNR of FABIAN (3 types, left ear) of Example 1.

[0036] Fig. 22This is a graph showing the comparison results of SNR of FABIAN (only four directions, right ear) of Example 1.

[0037] Fig.23 This is a graph showing the comparison results of SNR of FABIAN (only four directions, left ear) of Example 1.

[0038] Fig.24 This is a graph of integer multiple time shifts in the sound image panning of FABIAN (4 directions_diagonal, right ear) in Example 1.

[0039] Fig.25 This is a graph of integer multiple time shifts in the sound image panning of FABIAN (4 directions_diagonal, left ear) in Example 1.

[0040] Fig.26 This is a graph of integer multiple time shifts in the sound image panning of FABIAN (4 directions_vertical and horizontal, right ear) in Example 1.

[0041] Fig. 27 This is a graph of integer multiple time shifts in the sound image panning of FABIAN (4 directions_vertical and horizontal, left ear) in Example 1.

[0042] Fig.28 This is a graph of integer multiple time shifts in the sound image panning of FABIAN (6 directions, right ear) in Example 1.

[0043] Fig.29 This is a graph of integer multiple time shifts in the sound image panning of FABIAN (6 directions, left ear) in Example 1.

[0044] Fig.30 This is a graph showing the comparison results of the effect of fractional shifting of Example 1 verified using SNR (4 directions, diagonal direction, right ear).

[0045] Fig.31 This is a graph showing the comparison results of the effect of fractional shifting of Example 1 (4 directions, diagonal direction, left ear) verified using SNR.

[0046] Fig.32 This is a graph showing the comparison results of the effect of fractional shifting of Example 1 (four directions, vertical and horizontal, right ear) verified using SNR.

[0047] Fig.33 This is a graph showing the comparison results of the effect of fractional shifting of Example 1 (four directions, vertical and horizontal, and left ear) verified using SNR.

[0048] Fig.34This is a graph showing the comparison results of verifying the effect of fractional shifting in Example 1 using SNR (6 directions, right ear).

[0049] Fig.35 This is a graph showing the comparison results of verifying the effect of fractional shifting in Example 1 using SNR (6 directions, left ear).

[0050] Fig.36 This is an example of my HRIR waveform in comparison with Example 1.

[0051] Fig.37 This is an example of the FABIAN waveform of Comparative Example 1.

[0052] Fig.38 3 is a graph showing the waveform after frequency weighting of Comparative Example 2. DETAILED DESCRIPTION

[0053] The sound generating device of Example 1 is characterized in that it comprises: a direction acquisition unit for acquiring a sound source direction of a sound source; and a sound image shifting unit for performing sound image shifting of a sound from a specific representative direction by time shifting and gain adjustment of the sound source based on the sound source direction acquired by the direction acquisition unit, thereby representing the sound source.

[0054] The sound generating device of Example 2 may be the sound generating device of Example 1, characterized in that there are a plurality of the sound sources, the representative directions are directions for each representative point, and the number of the representative directions is less than the number of the sound sources, and the sound image translation unit synthesizes the sound images of the plurality of the sound sources through the sounds of the plurality of the representative directions.

[0055] The sound generating device of Example 3 may be the sound generating device of Example 2, characterized in that the sound image translation unit performs a time shift on the sound source in a manner that maximizes the cross-correlation between the head impulse response in the direction of the sound source and the head impulse response in the representative direction, or performs a time shift by adding a negative sign to the time shift.

[0056] The sound generating device of Example 4 may be the sound generating device of Example 3, characterized in that the time shift and / or gain is the result obtained by calculating the cross-correlation after applying weighted filtering on the frequency axis.

[0057] The sound generating device of Example 5 may be any one of Examples 2 to 4, characterized in that the sound image translation unit applies a gain set for each sound source and each representative direction to each of the multiple representative points for the time-shifted sound source.

[0058] The sound generating device of Example 6 can be any one of the sound generating devices of Examples 1 to 5, and is characterized in that, when the sound image translation unit synthesizes the HRIR vector of the sound source direction by the sum of the HRIR vectors representing the directions, it uses a gain calculated in such a way that the error signal vector between the synthesized HRIR vector and the HRIR vector of the sound source direction is orthogonal to the HRIR vector representing the direction.

[0059] The sound generating device of Example 7 can be any one of the sound generating devices of Examples 1 to 6, characterized in that the sound image translation unit uses a gain calculated in a manner that minimizes the energy or L2 norm of the error signal vector between the synthesized HRIR vector and the HRIR vector in the direction of the sound source.

[0060] The sound generating device of Example 8 may be the sound generating device of Example 6 or Example 7, characterized in that the error signal vector uses the result obtained by applying weighted filtering on the frequency axis.

[0061] The sound generating device of Example 9 can be any one of the sound generating devices of Examples 2 to 5, characterized in that the sound image translation unit uses a corrected gain so that the energy balance of the head impulse responses of the left ear and the right ear from the position of the sound source is also maintained in the head impulse response that is essentially synthesized by the head impulse responses from multiple representative points through sound image translation.

[0062] The sound generating device of Example 10 can be any one of the sound generating devices of Examples 4, 5, and 9, and is characterized in that the sound image translation unit performs the time shift on the sound source, and processes the signal obtained by applying the gain as a representative point signal existing at the position of the representative point, and convolves the head impulse response at the position of the representative point with the sum signal of the representative point signals of the number of the sound sources to generate a signal at the listener's ear.

[0063] The sound generating device of Example 11 can be any one of the sound generating devices of Examples 1 to 10, characterized in that the time shift also allows the shift of the decimal point of the sampling.

[0064] The sound generating device of Example 12 may be any one of the sound generating devices of Examples 1 to 11, characterized in that the tendency of high-frequency region attenuation is compensated by reproducing high-frequency region enhancement filtering.

[0065] The sound generating device of Example 13 can be any one of the sound generating devices of Examples 1 to 12, characterized in that the sound source is a sound signal of the content and a sound signal of a participant in a remote call, and the direction acquisition unit acquires the direction of the sound source observed from the listener.

[0066] The sound reproducing device of Example 14 is characterized by comprising: the sound generating device of any one of Examples 1 to 13; and a sound output unit that outputs the sound signal generated by the sound generating device.

[0067] The sound generation method of Example 15 is a sound generation method executed by a sound generating device, characterized in that the sound source direction of the sound source is obtained, and based on the obtained sound source direction, the sound image of the sound from a specific representative direction is shifted by time shifting and gain adjusting the sound source to express the sound source.

[0068] The sound signal processing program of Example 16 is a sound signal processing program executed by a sound generating device, characterized in that the sound signal processing program enables the sound generating device to: obtain the sound source direction of the sound source, and based on the obtained sound source direction, perform sound image translation of the sound from a specific representative direction through time shift and gain adjustment of the sound source, thereby expressing the sound source.

[0069] <First Embodiment>

[0070] [Control Structure of Sound Reproducing Device 1]

[0071] First, refer to Figure 1 , a control structure of the sound reproduction device 1 according to the first embodiment will be described.

[0072] The sound reproducing device 1 is a device that is worn by a listener and can reproduce sound. It reproduces an acoustic signal that is a content of data such as video, sound, and text, and can make a call with a remote location.

[0073] Specifically, the sound reproduction device 1 is, for example: a stereo reproduction device for a PC (Personal Computer) and a smartphone connected to a headset, a dedicated game console, a content reproduction device for reproducing content stored in optical media and flash memory cards, equipment for movie theaters and public audio-visual venues, a headset equipped with a dedicated decoder and a head tracking sensor, an HMD (Head-Mounted Display) for VR (Virtual Reality), AR (Augmented Reality) and MR (Mixed Reality), a headset-type smartphone, a video conferencing system, equipment for remote conferencing, auxiliary devices for listening to sound, hearing aids, and other home appliances.

[0074] The sound reproduction device 1 of this embodiment includes a direction acquisition unit 10, a sound image panning unit 20, an output unit 30, and a reproduction unit 40 as a control structure. In this embodiment, the direction acquisition unit 10 and the sound image panning unit 20 constitute a sound generation device 2 that generates a sound signal.

[0075] Here, in this embodiment, stereo sound is generated from the sound sources S-1 to Sn, which are multiple sound signals (sound source signals, target signals). Hereinafter, any one of the multiple sound sources S-1 to Sn is also referred to as "sound source S". As the sound source S in this embodiment, the sound signal of the content, the sound signal of the remote call participant, etc. can be used.

[0076] The content may be various contents such as games, movies, VR, AR, MR, etc. The movie may also include musical instrument performances, speeches, etc. In such a case, as the sound source S, it is possible to use sound signals from objects such as musical instruments, vehicles, game characters, etc. (hereinafter referred to as "objects, etc."), and sound signals from people such as actors, commentators, rakugo artists, orators, and other speakers who are the sound generation sources. These sound signals are set with spatial configuration relationships in terms of content.

[0077] Alternatively, in the case where the sound source S is a voice signal of a participant in a remote call, it is possible to use a voice signal emitted by a user (participant) of various messaging or video conferencing application software (Application Software, hereinafter referred to as "application") such as a PC (Personal Computer) and a smartphone. The voice signal can be obtained by a microphone of a head-mounted device, etc., or by fixing it to a table, etc. As direction information, the direction of the participant's head in the camera, or the direction of an avatar configured in a virtual space, etc. can be added. Furthermore, the sound source S can be a voice signal of a participant in a remote conference such as a video conferencing system between multiple bases in a one-to-one, one-to-many, or many-to-many manner. In such a case, the direction of the participants of each call relative to the camera can also be set as direction information.

[0078] In any case, as the sound source S, a sound signal recorded by a network or directly connected microphone or the like can also be used. In such a case, direction information can also be added to the sound signal. Alternatively, any combination of the above contents and the sound signal of the remote participant can be used. In addition, in the present embodiment, the sound signal of the sound source S also becomes a "target signal" for reproducing the direction of the stereo sound.

[0079] The direction acquisition unit 10 acquires the sound source direction of the sound source S. In the present embodiment, the direction acquisition unit 10 acquires the direction of the sound source S relative to the front direction of the listener. In addition, the direction acquisition unit 10 may acquire the direction of the listener relative to the radiation direction of the sound source S. Specifically, the direction acquisition unit 10 acquires the direction of the sound source S as viewed from the listener. In addition, the direction acquisition unit 10 may acquire the direction of the listener as viewed from the sound source S.

[0080] Here, in the sound source S of the present embodiment, the direction information when the sound is emitted is calculated or set. Therefore, the direction acquisition unit 10 acquires the radiation direction of the sound emitted by the sound source S. In the present embodiment, for example, the direction acquisition unit 10 can acquire the direction of the head of the participant that becomes the sound source S. In addition, the direction acquisition unit 10 can also acquire the direction of the listener's head based on the direction information such as the orientation of the avatar in the virtual space through head tracking based on a gyro sensor of an HMD or a smartphone.

[0081] The direction acquisition unit 10 can calculate the directions of the sound source S and the listener in the spatial arrangement including the virtual space based on the information of these directions.

[0082] The sound image panning unit 20 performs sound image panning of the sound from a specific representative direction by time shifting and gain adjustment of the sound source S based on the sound source directions of the plurality of sound sources S (target signals) acquired by the direction acquisition unit 10, thereby performing sound image panning for expressing the sound source S. Specifically, the sound image panning unit 20 synthesizes the sound source S (target signal) by sound image panning in a representative direction that is similar to the sound source direction of the sound source S. Thus, the sound image panning unit 20 equivalently generates the HRIR of the sound source direction of the sound source S. Here, in the present embodiment, as shown in the embodiments described later, "equivalent" and "equivalent" refer to signals that have an error of less than a specific degree and are substantially the same. Specifically, by panning the sound image of the sound source S, the sound image panning unit 20 synthesizes the HRIRs of several directions that are closest to the sound source direction of the sound source S or are most similar to the HRIR of the sound source direction, thereby equivalently generating the HRIR of the direction. In the present embodiment, the direction is described as a "specific representative direction" (hereinafter also referred to as "representative direction") described below. This reduces the amount of computation required to generate the ear signal.

[0083] That is, the sound image panning unit 20 synthesizes the sound images of the multiple sound sources S by using the sounds of the multiple representative directions. The representative directions can use, for example, 2 to 3 directions. Specifically, the sound image panning unit 20 can synthesize the sound image using only the HRIR of the representative directions for the representative points, the number of which is less than the number of the sound sources S.

[0084] At this time, the sound image panning unit 20 calculates a time shift (delay) that maximizes the cross-correlation between the HRIR in the sound source direction of the sound source S and the HRIR in the representative direction. The time shift obtained here or the time shift with a minus sign added to the time shift is given to the sound source S, and the obtained time-shifted signal is used as the signal in the representative direction for subsequent processing.

[0085] This time shift may allow a time shift shorter than the sampling frequency (a shift in sampling position expressed as a decimal, hereinafter referred to as "fractional shift"). This fractional shift can be performed by oversampling.

[0086] Here, the sound image translation unit 20 adds a gain to the signal of the representative direction after the sound source S is time-shifted, calculates the convolution of the HRIR of each representative point with the sum of these values ​​calculated for each representative point, and synthesizes a signal equivalent to the signal obtained by convolving the HRIR of the sound source direction with the sound source S.

[0087] On the other hand, the sound image panning unit 20 can calculate the gain by making the error signal vector between the synthesized HRIR (vector) and the HRIR (vector) in the sound source direction orthogonal to the HRIR (vector) in the representative direction when synthesizing the HRIR (vector) in the sound source direction by the sum of the HRIR (vector) in the representative direction. In addition, the HRIR (vector) refers to the time waveform of the HRIR as a vector. Hereinafter, the HRIR (vector) is also described as an "HRIR vector".

[0088] The sound image panning unit 20 corrects the gain so that the energy balance of the HRIR of the left ear and the right ear from the sound source position is maintained even in the HRIR substantially synthesized from the HRIRs from a plurality of representative points by sound image panning. That is, the sound image panning unit 20 can correct the gain so that the energy balance of the HRIR of the left ear and the right ear of the listener caused by the sound source S is maintained even in the HRIR substantially synthesized by sound image panning.

[0089] In this embodiment, the sound image panning unit 20 calculates a gain value representing the HRIR gain of each direction and a time shift value corresponding to the time of the HRIR time shift for each sound source direction of the sound source S, and can store them in advance in the HRIR table 200 described later.

[0090] On this basis, the sound image panning unit 20 performs time shift of each sound source S using the time shift value and gain value corresponding to the sound source direction of each sound source S, applies a gain, and takes the sum as a sum signal. The sound image panning unit 20 processes the sum signal as a signal existing at the position of the representative point. The sound image panning unit 20 can generate a signal at the ear of the listener by convolving the HRIR of the position of the representative point with the sum signal.

[0091] The output unit 30 outputs the sound signal generated by the sound generating device 2. In the present embodiment, the output unit 30 includes, for example, a D / A converter, an amplifier for headphones, etc., and outputs the sound signal as a reproduced sound signal for headphones, i.e., the reproduction unit 40. Here, the reproduced sound signal may be, for example, a sound signal that can be heard by a listener by decoding digital data based on information included in the content and reproducing it in the reproduction unit 40. Alternatively, the output unit 30 may reproduce the sound signal by encoding the sound signal and outputting it as a sound file or streaming sound.

[0092] The reproduction unit 40 reproduces the reproduction sound signal output by the output unit 30. The reproduction unit 40 may include a speaker including an electromagnetic driver and a diaphragm such as headphones or earphones (hereinafter referred to as a "speaker, etc."), an earmuff or receiver worn by a listener, or the like.

[0093] Alternatively, the reproduction unit 40 may directly output the digital reproduction sound signal from a speaker or the like as a digital signal, or convert the digital reproduction sound signal into an analog sound signal using a D / A converter and output it from a speaker or the like, so that the listener can hear it. Alternatively, the reproduction unit 40 may also output the sound signal to headphones or earphones of an HMD worn by the listener.

[0094] The HRIR table 200 is HRIR data of a representative point selected by the sound image panning unit 20. The HRIR table 200 includes various values ​​used for synthesis of sound image panning using HRIR, which are calculated by the sound image panning unit 20 described later.

[0095] Specifically, the HRIR table 200 includes, for example, gain values ​​calculated for each representative point in the sound source direction at every 2° on a 360° circumference as the above-mentioned values. Regarding the above-mentioned gain values, for example, when performing sound image panning in two directions of left and right with two representative points, two gain values ​​(A value and B value) can be used for each sound source direction, and when performing sound image panning in three directions including the elevation direction, three gain values ​​(A value, B value and C value) can be used.

[0096] In addition, the HRIR table 200 may further include a time shift value for time-shifting the sound source S. The time shift value may include a fractional shift value for performing a fractional shift by oversampling the sound source S. The HRIR table 200 may store the time shift value in correspondence with the gain value.

[0097] These gain values ​​and time shift values ​​can be calculated off-line in advance.

[0098] [Hardware Configuration of Sound Reproducing Device 1]

[0099] The sound reproduction device 1 includes, for example, a control unit (control part) such as an ASIC (Application Specific Processor), a DSP (Digital Signal Processor), a CPU (Central Processing Unit), an MPU (Micro Processing Unit) and a GPU (Graphics Processing Unit) as various circuits.

[0100] In addition, the sound reproduction device 1 may include semiconductor memories such as ROM (Read Only Memory) and RAM (Random Access Memory), magnetic recording media such as HDD (Hard Disk Drive), optical recording media and other storage units as storage units (storage units). As ROM, flash memory and other writable and recordable recording media may be included. In addition, an SSD (Solid State Drive) may be provided instead of the HDD. The storage unit may store the control program and various contents of the present embodiment. Among them, the control program is a program for implementing various functional structures and methods of the sound signal processing program including the present embodiment. The control program includes embedded programs such as firmware, OS (Operating System), and applications.

[0101] The various contents may be, for example: data of movies and music; games; audiobooks; data of electronic books capable of sound synthesis; broadcast data of television and radio; various sound data related to operation instructions of car navigation systems and various home appliances, etc.; entertainment content including VR, AR, MR, etc.; and other data capable of outputting sound. Alternatively, the content may be the BGM or sound effects of a game, MIDI (Musical Instrument Digital Interface) files, voice call data from a mobile phone or transceiver, etc., or data of synthesized sound of text in a message. These contents may be obtained by downloading files or data blocks transmitted by wire or wirelessly, or by obtaining them in stages by streaming, etc.

[0102] Furthermore, the application of the present embodiment may be an application such as a media player that reproduces content, an application for messaging or video conferencing, or the like.

[0103] In addition, the sound reproduction device 1 can have a direction calculation unit, which includes: a GNSS (Global Navigation Satellite System) receiver for calculating the direction the listener is facing, an indoor position direction detector, an acceleration sensor capable of head tracking, a gyro sensor, a geomagnetic sensor, etc.; and a circuit for converting their outputs into direction information.

[0104] In addition, the sound reproduction device 1 may include: a display unit such as a liquid crystal display or an organic EL display; an input unit such as a button, a keyboard, a pointing device such as a mouse or a touch panel; and an interface unit for connecting to various devices by wireless or wired. The interface unit may include: an interface for a flash memory medium such as a micro SD (registered trademark) card or a USB (Universal Serial Bus) memory; and an interface for a LAN board, a wireless LAN board, a serial interface, a parallel interface, and the like.

[0105] In addition, the sound reproduction device 1 can realize the methods of the present embodiment using hardware resources by making the control unit execute the methods of the present embodiment using various programs mainly stored in the storage unit. In addition, a part of the above structure or any combination thereof can be constituted in hardware or circuit by IC, programmable logic, FPGA (Field-Programmable Gate Array), etc.

[0106] [Sound Reproduction Processing of Sound Reproducing Device 1]

[0107] Next, refer to Figure 2 to Figure 4 Next, a description is given of the sound reproduction process of the sound reproduction device 1 according to the first embodiment.

[0108] First, through Figure 2 The outline of the sound reproduction process of this embodiment will be described.

[0109] In order to generate the sound emitted from the sound source S at the ear, the HRIR (Head Impulse Response) is convolved with each sound source S and the results are added, wherein the HRIR represents the transfer function from each sound source direction to the ear of the left and right ears on the time axis, that is, the head transfer function (HRTF). Figure 2 , an example of convolving HRTFs for sound sources S-1, S-2, S-3, and S-4 is shown.

[0110] However, in this method, as the number of sound sources S increases, the amount of convolution operations for performing a large number of product-sum operations increases.

[0111] In contrast, in the sound reproduction process of the present embodiment, instead of directly convolving the HRIR from each sound source S to the ear with each sound source S, each sound source S is synthesized and expressed by panning the sound image of representative points R-1 to Rn (hereinafter, when one of these representative points is indicated, simply referred to as "representative point R"), and convolution of the HRIR from the representative point R to the ear is performed. As a result, it is possible to express a sound image based on stereo sound as if all the sound sources S are reproduced at the ear. As a result, even if the number of sound sources S increases, since the number of convolutions is determined only by the number of representative points, the operation for convolution does not increase.

[0112] exist Figure 2 In the example, the sound sources S-1 to S-4 are represented by the sound image shift between the representative point R-1 and the representative point R-2. Although there are four sound sources, only two of them, the representative points R-1 and R-2, are convolved. In addition, for the background, the representative points R-3 and R-4 can be added to perform sound image shift.

[0113] In this embodiment, when the sound image shifting unit 20 performs sound image shifting, the sound source S (target signal) may be time-shifted, and a signal obtained by applying a gain to the time-shifted sound source S may be processed as a representative point signal existing at the position of the representative point R. On this basis, the sound image shifting unit 20 calculates a sum signal of representative point signals of the number of sound sources S concentrated on the representative point R, and convolves the HRIR at the position of the representative point with the sum signal to generate a signal at the ear of the listener U.

[0114] That is, if there are n sound sources S using one representative point R, the sound image panning unit 20 can generate an ear signal by convolving the result of adding the HRIR of the position of the representative point R with the representative point signals of the n sound sources S.

[0115] The sound reproduction processing of this embodiment is mainly performed in the sound reproduction device 1, and can be implemented by the control unit cooperating with each part and using hardware resources to control and execute the control program stored in the storage unit, or can be directly executed by the circuit.

[0116] Below, refer to Figure 3 The flowchart explains the details of the sound reproduction process step by step.

[0117] (Step S101)

[0118] First, the direction acquisition unit 10 of the sound reproduction device 1 performs a sound source and direction acquisition process. The direction acquisition unit 10 acquires the direction of the sound source S viewed from the listener U.

[0119] Specifically, the direction acquisition unit 10 acquires a sound signal (target signal) of the sound source S. The sampling frequency and quantization bit number of the sound signal are arbitrary. In this embodiment, an example of using a sound signal with a sampling frequency of 48kHz and a quantization bit number of 16 bits is described. In addition, the direction acquisition unit 10 acquires the direction information of the sound source S, such as the sound signal attached to the content or the sound signal of the participant of the remote call.

[0120] On this basis, the direction acquisition unit 10 grasps the spatial configuration of the sound source S and the listener U. As described above, the configuration may be a configuration in a space including a virtual space set in the content, etc. Then, the direction acquisition unit 10 calculates the direction of the sound source S as viewed from the listener U, that is, the sound source direction, based on the grasped configuration in the space. The same is true for the sound signal of the content. The direction acquisition unit 10 can refer to the direction information of the sound signal of the sound source S and calculate the sound source direction based on the configuration of the listener U.

[0121] Furthermore, the direction acquisition unit 10 may also calculate the direction of the listener U viewed from the sound source S.

[0122] (Step S102)

[0123] Next, the sound image panning unit 20 performs sound image panning processing. Here, the sound image panning unit 20 uses the direction information to perform sound image panning of the sound source S. In this embodiment, the sound image panning unit 20 performs sound image panning based on the viewpoint of how the sound synthesized near the ear through sound image panning can be close to the sound that should originally be near the ear.

[0124] according to Figure 4 , the operation performed by the sound image panning unit 20 when panning the sound source S-1 using the representative points R-1 and R-2 is described. Figure 4 Shows Figure 2 Here, although the signal for sound image panning is the sound source S-1, in order to calculate the required optimal shift amount and optimal gain, the sound source S-1, the representative point R-1, and the HRIR from the representative point R-2 to the ear are used for calculation.

[0125] In this Figure 4 In the example of , the HRIR with the number of sampling points (tap number) from the sound source S-1 to the ear is set as a P-dimensional vector. It is set as v{x} (in each of the following embodiments, the vector is represented by "v{}").

[0126] Here, the sound image panning unit 20 sets the HRIR from the representative point R-1 to the ear of the listener U as v{x 01}, and the HRIR from the representative point R-2 to the ear is set to v{x 02}. Calculate v{x} and v{x 01}, and will be used to maximize the cross-correlation between v{x 01} is time-shifted vector v{x1}. Similarly, calculate v{x} and v{x 02}, and will be used to maximize the cross-correlation between v{x 02}The vector after time shift is calculated as v{x2}.

[0127] Gain A is applied to v{x1}, gain B is applied to v{x2}, and v{x} is approximated by the sum of these. That is, v{x} is approximated by setting the approximate value of v{x} = A×v{x1}+B×v{x2}. This makes it possible to achieve panning with less error.

[0128] The calculation of the gain and the time shift will be described in detail. First, the calculation of the gain will be described. The error vector generated by the approximation of v{x} is represented by the following equation (1).

[0129] [Mathematical formula 1]

[0130]

[0131] In the above formula (1), the arrow on the variable indicates that it is a vector. Here, when A and B have the optimal size, that is, when the size of the error vector is the smallest, the error vector v{e} is orthogonal to the plane formed by the vectors v{x1} and v{x2} of the synthesis source. Therefore, the relationship of the following formula (2) holds.

[0132] [Mathematical formula 2]

[0133]

[0134] Thus, the following formula (3) is calculated.

[0135] [Mathematical formula 3]

[0136]

[0137] When the formula (3) is transformed, the following formula (4) is obtained.

[0138] [Formula 4]

[0139]

[0140] If |v{x2}| 2 By performing the operation of v{x1}·v{x2} on the following formula, we can obtain the following formula (5).

[0141] [Formula 5]

[0142]

[0143] A can be calculated by subtracting the following equation from the above equation (5) and eliminating B. This is expressed in equation (6).

[0144] [Mathematical formula 6]

[0145]

[0146] Therefore, the gain A becomes the following formula (7).

[0147] [Mathematical formula 7]

[0148]

[0149] Similarly, by eliminating the gain A, the gain B can be calculated as shown in the following equation (8).

[0150] [Mathematical formula 8]

[0151]

[0152] In this way, the gains A and B are determined so that the error vector between the synthesized signal and the target signal is orthogonal to the representative direction vector used.

[0153] By applying the gains A and B obtained by this calculation to the HRIR waveform of v{x1} and the HRIR waveform of v{x2} after the time shift based on the cross-correlation, the HRIR to be output can be synthesized. That is, these time shift amounts (time shift values) and gains A and B are applied to the sound source S-1 to perform sound image panning.

[0154] Next, the specific calculation process of the time shift that maximizes the cross-correlation is described. 01} The HRIR with P sample points is treated as a vector. Therefore, the subscript of the HRIR time (the position of the sample point) can be clearly expressed as the following formula (9).

[0155] [Mathematical formula 9]

[0156]

[0157] On this basis, the cross-correlation between the two vectors in the equation (9) is defined as a function of "k" as shown in the following equation (10).

[0158] [Formula 10]

[0159]

[0160] Here, given φ xx01 The maximum value of (k) is denoted by k max01The sound image panning unit 20 calculates k by substituting each value into k, for example. max01 Similarly, given φ xx02 The maximum value of (k) is denoted by k max02 The sound image translation unit 20 is connected with k max01 The k is calculated in the same way max02 . The following will be k max01 or k max02 One of them is abbreviated as "k max ”.

[0161] For example, the sound image panning unit 20 calculates the gains A, B, and k for the sound source directions of the sound sources S that are different every 2° on a 360° circumference. max01 , k max02 The gain values ​​A and B and the time shift values ​​are stored in advance in the HRIR table 200 and used in the output process described below. max01 , k max02 The HRIR table 200 calculated based on the value of , only performs the following sound output processing.

[0162] (Step S103)

[0163] Next, the sound image shifting unit 20 and the output unit 30 perform sound output processing. First, the sound image shifting unit 20 obtains the gain value and time shift value corresponding to the acquired sound source direction from the HRIR table 200 for each sound source S. On this basis, the sound image shifting unit 20 applies the gain value to each sampling point (sample) of the waveform of the sound source S.

[0164] At this time, the sound image panning unit 20 can correct the gain so that even in the HRIR synthesized by the sound image panning, the energy balance of the HRIR of the left ear and the right ear caused by the sound source S is maintained. In other words, an adjustment coefficient that makes the energy balance between the left HRIR and the right HRIR consistent with the original HRIR can be applied to each gain value.

[0165] Next, the sound image panning unit 20 time-shifts the signal to which the gain value is applied.

[0166] The details of this time shift are described below. 01} is shifted by k max The vector v{x1} of samples.

[0167] First, in the case of phase advance, that is, in k max ≥0, set k at the end of the vector max On the other hand, in the case of phase delay, i.e., in k max<0, set k at the beginning of the vector max The length of the vector is maintained by setting the zero of samples. That is, it is set as shown in the following equation (11).

[0168] [Mathematical formula 11]

[0169]

[0170] In this way, a time-shifted vector v{x1} is generated. The positive and negative polarity of the time shift value is reversed depending on the reference when calculating the above cross-correlation. In addition, when convolving the HRIR with the sound source signal, it is necessary to pay attention to the polarity of the time shift.

[0171] In addition, as shown in the embodiment described later, the sound image shifting unit 20 can also perform the time shift by oversampling to perform a decimal shift by a decimal multiple, rather than performing the time shift by an integral multiple of the number of taps. Alternatively, a gain value may be applied after the time shift.

[0172] The sound image panning unit 20 processes the signal calculated in this way and subjected to gain and time shift as a representative point signal existing at the position of the representative point R. On this basis, the sound image panning unit 20 sums the representative point signals of the sound source S concentrated on the representative point R to generate a sum signal. Then, the sound image panning unit 20 convolves the HRIR at the position of the representative point R (HRIR in the representative point direction) with the sum signal to generate a signal at the ear of the listener U.

[0173] The output section 30 reproduces the ear signal generated by the sound image panning section 20 by outputting it to the reproduction section 40. The output may be, for example, a two-channel analog sound signal corresponding to the left and right ears of the listener U.

[0174] Thus, the reproduction unit 40 can reproduce the sound signal corresponding to the virtual sound field as a two-channel sound signal of the headphone. As described above, the sound reproduction process of the first embodiment ends.

[0175] The above-described structure can provide the following effects.

[0176] In recent years, when using VR headsets or HMDs to reproduce content such as movies, AR, VR, MR, games, etc., it is necessary to appropriately describe and reproduce the rendering technology (binaural technology) of the entire 3D sound field. In the generation of existing 3D stereo sound (binaural signals), multiple sound source signals are generated by convolving the HRIR of the corresponding sound source directions separately. In this way, when the HRIR is convolved with each sound source S, a large amount of calculation is required to follow the movement of the person with a high sense of presence (6DoF: 6 degrees of freedom), which has become a problem.

[0177] On the other hand, in the case of sound image panning by speakers, sound images are conventionally created between speakers by controlling the volume balance of the speakers using the sine law, the tangent law, etc. However, it is not possible to appropriately reproduce the stereo sound image of the headphone simply by controlling the volume balance.

[0178] In contrast, the sound generating device 2 of the representative example (A) is characterized in that it comprises: a direction acquisition unit 10 for acquiring the sound source direction of the sound source S; and a sound image translation unit 20 for performing sound image translation of the sound from a specific representative direction by time shifting and gain adjusting the sound source S based on the sound source direction acquired by the direction acquisition unit 10, thereby expressing the sound source S.

[0179] With such a structure, more efficient and effective rendering can be achieved by synthesizing the sound source S through sound image panning of representative directions and reducing the number of sound source directions. As a result, the amount of calculation can be reduced compared to the existing method of convolving the HRIR with the signal of each sound source S individually. That is, the sound image panning unit 20 can equivalently synthesize the HRIR of the representative direction that is similar to the sound source direction obtained by the direction acquisition unit 10 through sound image panning, and generate the HRIR of the sound source direction. By reducing the amount of calculation in this way, it can be applied to VR / AR applications such as games and movies as a reproduction system of a 3D sound field. In addition, by applying to smartphones and home appliances, the amount of calculation for generating stereo sound can be suppressed, and costs can be reduced. In addition, as a way to further reduce the amount of calculation, it can be applied to international standardization, etc.

[0180] In addition, the sound generating device 2 of the representative example (B) is the sound generating device 2 of the representative example (A), and is characterized in that there are multiple sound sources S, the number of representative directions is less than the number of sound sources S, and for the direction of each representative point, the sound image translation unit 20 synthesizes the sound images of the multiple sound sources S through the sounds of the multiple representative directions.

[0181] With such a configuration, the sound sources S located in a plurality of sound source directions are panned in predetermined representative directions, such as 2 to 6 directions around the listener U, and the sound sources S are gathered in these directions and then convolved with the HRIR. This can reduce the amount of calculation compared to the conventional method of convolving the HRIR with each sound source signal individually.

[0182] In addition, the sound generating device 2 of representative example (C) is the sound generating device 2 of representative example (A) or (B), and is characterized in that the sound image translation unit 20 performs a time shift on the sound source S in a manner that maximizes the cross-correlation between the HRIR in the direction of the sound source S and the HRIR in the representative direction, or performs a time shift in which a negative sign is added to the time shift.

[0183] With such a configuration, the sound image panning unit 20 calculates a time shift amount (time shift value) for each sound source direction so as to maximize the cross-correlation between the HRIR in the sound source direction and the HRIR in the representative direction, applies the time shift amount (time shift value) to the sound source signal, and further multiplies it by an appropriate gain, thereby distributing the sound source signal to each representative direction. Thus, when performing sound image panning, the signal of the sound source S is time-shifted to suppress the distortion of the HRIR virtually synthesized by the sound emitted from the representative direction, and a signal can be generated by convolving the HRIR equivalent to the target HRIR with the sound source S. That is, the sound synthesized at the ear by time-shifting the sound source S and performing sound image panning can be made close to the sound at the ear generated by convolving a plurality of sound sources with the original HRIR.

[0184] The sound generating device 2 of the representative example (D) is any of the sound generating devices 2 of the representative examples (A) to (C), and is characterized in that the time shift also allows shifting of the decimal point of the sampling.

[0185] This structure enables panning with further reduced distortion. That is, as shown in the embodiments described below, it is possible to suppress the change in the comb shape of the S / N ratio (Signal-Noise Ratio, hereinafter referred to as "SNR") due to integer shifting and improve the SNR.

[0186] In addition, the sound generating device 2 of representative example (E) is any one of the sound generating devices 2 in representative examples (A) to (D), characterized in that the sound image translation unit 20 applies a gain set for each sound source S and each representative direction to the time-shifted sound source S to each of the plurality of representative points.

[0187] With such a configuration, a gain set for each sound source S is applied to each representative point R, and the sum of the signals after the gain is applied to all the sound sources S is calculated. That is, the sound image panning unit 20 applies a gain to the time-shifted sound source S, and convolves the result of calculating the sum with the HRIR of the representative direction, thereby equivalently synthesizing a signal obtained by convolving the HRIR of the sound source direction with the sound source S. This makes it possible to minimize distortion in the sound image panning, reduce the amount of calculation, and reproduce stereo sound based on HRIR.

[0188] In addition, the sound generating device 2 of representative example (F) is any one of the sound generating devices 2 in representative examples (A) to (E), and is characterized in that when the sound image translation unit 20 synthesizes the HRIR (vector) of the sound source direction by the sum of the HRIR (vector) of the representative directions, it uses a gain calculated so that the error signal vector between the synthesized HRIR (vector) and the HRIR (vector) of the sound source direction is orthogonal to the HRIR (vector) of the representative direction.

[0189] With such a structure, when the HRIR (vector) in the direction of the sound source is synthesized by the sum of the HRIR (vector) representing the direction, the error signal vector between the synthesized HRIR (vector) and the HRIR (vector) in the direction of the sound source is made orthogonal to the HRIR (vector) representing the direction, and the gain is calculated. That is, sound image panning is performed by calculating the gain that makes the equivalent synthesized HRIR most similar in shape to the original HRIR. Thus, in theory, sound image panning that minimizes distortion becomes possible. Therefore, it becomes possible to save computing resources and perform sound image panning suitable for listening to headphones such as AR / VR with higher accuracy than the sine law, tangent law, etc.

[0190] In addition, the sound generating device 2 of representative example (G) is any one of the sound generating devices 2 in representative examples (A) to (F), characterized in that the sound image translation unit 20 uses a corrected gain so that the energy balance of the HRIRs of the left ear and the right ear from the position of the sound source S is also maintained in the HRIR that is essentially a synthesis of HRIRs from multiple representative points through sound image translation.

[0191] With such a structure, the energy balance can be prevented from becoming unnatural through the synthesis of HRIR.

[0192] In addition, the sound generating device 2 of representative example (H) is any one of the sound generating devices 2 in representative examples (A) to (G), and is characterized in that the sound image translation unit 20 time-shifts the sound source S, and processes the signal obtained by applying a gain as a representative point signal existing at the position of the representative point, and convolves the HRIR of the position of the representative point with the sum signal of the representative point signals of the number of sound sources S to generate a signal at the ear of the listener U.

[0193] With such a configuration, it is possible to generate a high-quality stereo signal while suppressing the amount of computation. In addition, by calculating the gain value and the time shift value and storing them in advance in the HRIR table 200, applying these values ​​to the sound source S and calculating the sum signal, and convolving the HRIR of the position of the representative point with the sum signal, it is possible to reproduce the stereo sound. As shown in the embodiment described later, as the number of sound sources S increases, the computation load can be significantly reduced. Specifically, even if the number of sound sources S is 3 to 4, the number of product sum operations can be reduced to 65% to 80%.

[0194] In addition, the sound generating device 2 of representative example (I) is any one of the sound generating devices 2 in representative examples (A) to (H), characterized in that the sound source S is one of the sound signals of the content and the sound signals of the participants in the remote call, and the direction acquisition unit 10 acquires the direction of the radiation direction of the sound emitted by the listener U relative to the sound source S.

[0195] With such a structure, when reproducing content, it is possible to reduce the load and generate sound for a large number of sound sources S such as messages, remote conferences, etc. in one-to-one connection, one-to-multipoint connection, and multipoint-to-multipoint connection.

[0196] The sound reproducing device 1 of the representative example (J) is characterized by comprising: the sound generating device 2 of any one of the above (A) to (I); and a sound output unit 30 that outputs the sound signal generated by the sound generating device 2 .

[0197] With such a structure, the generated sound can be output through headphones, HMD, etc., and the sound with a sense of presence can be experienced.

[0198] In addition, in the above embodiment, when the sound image panning unit 20 represents the sound source signal by using the sound image panning of the representative points in the left and right directions, an example is described in which the HRIR vector in the sound source direction is equivalently synthesized using the HRIR vector in the left and right directions. That is, in the above embodiment, an example is described in which the left and right angle direction of the listener U is considered as the direction information.

[0199] However, the up and down directions may also be considered as these arrival directions. Specifically, the HRIR vector in the direction of the sound source can also be equivalently synthesized by interpolating the HRIR vectors in the three directions. That is, the sound image panning unit 20 can also perform sound image panning processing caused by representative points in the three directions including the elevation direction.

[0200] In this case, similarly to interpolation from two directions, the HRIR representing the direction is time-shifted so that the cross-correlation with v{x} is maximized, and is described as vectors v{x1}, v{x2}, and v{x3}. In this case, the error vector v{e} is represented by the following equation (12).

[0201] [Mathematical formula 12]

[0202]

[0203] Substitute this into the following equation (13) and solve it.

[0204] [Mathematical formula 13]

[0205]

[0206] Specifically, the optimal gains A, B, and C can be calculated using the following formula (14).

[0207] [Mathematical formula 14]

[0208]

[0209] Here, in the above equation (14), the "-1" at the upper right corner of the matrix means the inverse matrix. The time shift amount k of the HRIR of the representative direction determined so as to maximize the cross-correlation is max01 , k max02 , k max03 , is also calculated before the above-mentioned gain value in the same way as the value in the case of two directions.

[0210] Furthermore, in the above-described embodiment, an example in which two to four representative points R are used has been described.

[0211] However, of course, more than two representative points R may be used. For example, as shown in the embodiment described later, 4 to 6 representative points R corresponding to range angles of 90°, 60°, etc. may be used. In addition, in the case of 4 representative points, the positions of different representative points R may be set so that they are oblique (45°, 135°, 225°, 315°), vertical and horizontal (0°, 90°, 180°, 270°) relative to the listener U. It is also possible to select two or three points closest to the direction of the sound source from the 4 to 6 representative points R and use them as representative points R for synthesizing the sound source.

[0212] That is, the sound generating device 2 of representative example (K) is any one of the sound generating devices 2 in representative examples (A) to (H), characterized in that the sound image translation unit 20 uses a gain calculated in a manner that minimizes the energy or L2 norm of the error signal vector between the synthesized HRIR vector and the HRIR vector in the direction of the sound source.

[0213] Furthermore, the sound reproducing device 1 of the representative example (L) may include: the sound generating device 2 of the representative example (K); and a sound output unit 30 that outputs a sound signal generated by the sound generating device 2 .

[0214] With such a structure, the HRIR vector in the direction of the sound source can be equivalently synthesized by interpolating the HRIR vectors in three directions.

[0215] <Second Embodiment>

[0216] (Weighted filtering when calculating time shift and gain)

[0217] In the first embodiment described above, an example of using HRIR itself when calculating the time shift and gain that maximizes the cross-correlation is described. However, in the sound generating device of the second embodiment, the time shift and / or gain can use the result obtained by calculating the cross-correlation after applying weighted filtering on the frequency axis. That is, when calculating the time shift and gain that maximizes the cross-correlation, the result obtained by applying weighted filtering on the frequency axis (hereinafter also referred to as "frequency weighted filtering") can be used.

[0218] The frequency weighted filter preferably uses a filter that uses a frequency band near or slightly above the frequency band where human hearing sensitivity is high as a cutoff frequency, and attenuates a band higher than the cutoff frequency, i.e., a band where human hearing sensitivity decreases. For example, a low-pass filter (LPF) with a cutoff frequency of 3000 Hz to 6000 Hz and an attenuation frequency of about 6 dB / Oct (octave) to 12 dB / Oct is preferably used.

[0219] Specifically, v{x} and v{x 01} The HRIR at point P is treated as a vector, so the time subscript of the HRIR is clearly marked, which can be marked as in the above equation (9). Here, as shown in the following equation (15), the impulse response w of the frequency weighted filter is c (n) is convolved with the two vectors of the above equation (9) and the length is truncated at P.

[0220] [Mathematical formula 15]

[0221] x w (n) = x(n)*w c (n), x 01w (n) = x 01 (n)*w c (n)……Formula (15)

[0222] Here, the operation "*" represents convolution. On this basis, the cross-correlation between two vectors in equation (15) is defined as a function of "k" as shown in the following equation (16).

[0223] [Formula 16]

[0224]

[0225] Here, we will give φ of equation (16) xx01 The maximum value of (k) is denoted by k max The sound image panning unit 20 generates the vector v{x 01} is shifted by k max The vector v{x1} after samples.

[0226] Specifically, in the case of phase advance, that is, in k max ≥0, fill the end of the vector with zeros to maintain the length of the vector to match k max That is, in k max ≥0, the vector v{x1} is v{x1}=(x 01 (0+k max ), x 01 (1+k max ), x01 (2+k max ), ... x 01 (P-1),…0,0,0).

[0227] On the other hand, in the case of phase delay, that is, in k max <0, fill the beginning of the vector with zeros to maintain the length of the vector to match k max That is, in k max <0, the vector v{x1} is v{x1}=(0, 0, 0, ..., x 01 (0), x 01 (1), x 01 (2), ..., x 01 (P-1+k max )).

[0228] In the above, the vector v{x 01w} can be used as a vector v{x 01}. In this way, the vector v{x1} can be generated. That is, similarly to the first embodiment, the cross-correlation can be calculated and used for calculating the time shift.

[0229] (Weighted filtering when calculating error)

[0230] In the first embodiment, when calculating the error (similarity) between the synthesized HRIR and the original HRIR, as shown in the above equation (12), the error signal vector (error vector) v{e} is calculated so that |v{e}| 2 Minimize A, B, C.

[0231] In this regard, in this embodiment, v{e} can use an object to which frequency weighted filtering is applied. Specifically, when v{e} is waveform data on the time axis, the result obtained by convolving the impulse response w(n) of the weighted filter with v{e} is set as v{e w}, then v{e w} is shown by the following formula (17).

[0232] [Mathematical formula 17]

[0233]

[0234] here, It is obtained by marking the impulse response w(n) as a vector.

[0235] The operation "*" indicates convolution. Here, the operator "*" is used for vectors, which marks the vectors on the left and right of the operator as series and then convolves them with each other, and marks the series obtained after the convolution with a vector. In other words, v{x}*v{y} marks the result of x(n)*y(n) as a vector. In the following, unless otherwise specified, the operator "*" for vectors is handled in the same way.

[0236] On this basis, it is possible to w}Substitute into the following equation (18) and solve it to calculate the gains A, B, and C.

[0237] [Mathematical formula 18]

[0238]

[0239] Alternatively, v{e}w can also be equivalently calculated using the following formula (19).

[0240] [Mathematical formula 19]

[0241]

[0242] here, It is obtained by marking the impulse response w(n) as a vector.

[0243] Using the time shift and gain found in this way, the target signal can be distributed (panned) in a representative direction.

[0244] In addition, the target signal and the convolution HRIR of the panning may be the same as those in the first embodiment, that is, the weighted filter may not be convolved with the target signal and the convolution HRIR.

[0245] By introducing such frequency weighting, the frequency band for approximation can be set with smaller error (higher accuracy). In particular, since the main energy of music and voice signals is concentrated in the low-frequency region, good performance can be obtained by using weighted filtering that gives weight to the low-frequency side.

[0246] In addition, if the convolution matrix W is used to represent the convolution of a weighted filter with an impulse response of w(n) and a vector, where each row of the convolution matrix W has the result of time-shifting the impulse response w(n) of the weighted filter sample by sample, then equation (17) can also be transformed into the following equation (20).

[0247] [Mathematical formula 20]

[0248]

[0249] On this basis, the following formula (21) can be used to calculate |v{e}| 2 .

[0250] [Mathematical formula 21]

[0251]

[0252] Here, W T Represents the transposed matrix of W.

[0253] In addition, when calculating the cross-correlation and when calculating the gain, a weighted filter with the same characteristics may be used, or a weighted filter with different characteristics may be used. When the same weighted filter is used, the weighted filter w may be convolved with the entire set of original HRIRs, and then the time shift and gain may be calculated using the same processing as in the first embodiment described above.

[0254] In addition, when using LPF to weight the low frequency region and calculate the optimal gain of the correlation as described above, the decimal shift of the first embodiment may not be performed when the effective band is limited to about 3000 Hz. In such a case, oversampling is not required.

[0255] (High frequency area enhancement filter)

[0256] In the above-mentioned embodiments, the sound signal image is panned and distributed in the representative directions of multiple directions, and the HRIR of each representative direction is represented by convolution. Specifically, in the above-mentioned first and second embodiments, the approximate value of v{x} in three directions is set to be A×v{x1}+B×v{x2}+C×v{x3}, and the sum of the HRIRs of the representative directions is used to simulate the HRIR of the target direction.

[0257] In this case, the amplitude characteristics of the high frequency region of HRIR tend to be lower than the original HRIR in comparison with the low frequency region. This is because even a slight time error caused by a slight shift in the position of the listening point will cause the phase of the high frequency region component of HRIR to rotate significantly, and the tendency of additive cancellation caused by the sound image shift will become stronger.

[0258] In contrast, in the sound generating device of the present embodiment, the tendency of high-frequency region attenuation can be compensated by reproducing the high-frequency region enhancement filter.

[0259] Specifically, by applying a high-frequency region enhancement filter to the signal of the HRIR representative direction panned and convolved with the sound image, the tendency of the high-frequency region to attenuate can be compensated. Alternatively, equivalently, a high-frequency region enhancement filter process can be applied to the representative direction HRIR itself in advance to enhance the high-frequency region. The high-frequency region enhancement filter can be, for example, a weighted filter with a turnover frequency of 5000 to 15000 Hz or more and an impulse response of the high-frequency region enhanced by about +1 to +1.5 dB.

[0260] In this way, by performing a filtering process to enhance the high-frequency region of the synthesized sound using sound image panning, the stereoscopic sense in the auditory sense can be further improved.

[0261] Even when the same fractional shift as in the first embodiment is performed, since mismatching of high-frequency region components of HRIR still occurs in normal 8 to 16 times oversampling, a high-frequency region enhancement filter may be applied.

[0262] [Other embodiments]

[0263] In the above embodiment, the case where the HRIR is convolved with the sound signal of the sound source S is described, but the same processing can be performed by converting the sound signal of the sound source S into the frequency domain and applying HRTF. In this case, different HRTFs can be applied to each frequency domain. Specifically, similar to the above second embodiment, by using HRTFs of the low and high frequency ranges based on the frequency band near or slightly higher than the frequency band with high sensitivity of human hearing, more accurate synthesis can be performed.

[0264] In addition, the sound image panning unit 20 may be capable of selecting a user's personal HRIR, an HRIR generated from an HRIR database, etc. from the HRIR table 200. Furthermore, when the speaker and the listener U are transformed into avatars in a virtual space, the sound image panning unit 20 may select an HRIR from the HRIR table 200 accordingly. That is, for example, when the avatar is in a shape such as a cat or a rabbit with ears located at the top, an HRIR that matches the listening style of the avatar can be selected.

[0265] In addition, the sound image panning unit 20 can further enhance the sense of reality by separating the direct sound of the sound source S and the reflected sound caused by the environment and superimposing them by convolution or the like. With such a configuration, a clear reproduced sound closer to reality can be reproduced.

[0266] In the above embodiment, an example has been described in which the reproduction unit 40 reproduces the audio using two left and right channels. However, the reproduction can also be performed using headphones or the like that can reproduce a plurality of channels.

[0267] Furthermore, in the above-mentioned embodiment, it is described that the sound reproducing device 1 is configured integrally.

[0268] However, the sound reproduction device 1 can be configured as a reproduction system that connects an information processing device such as a smartphone, a PC, or a home appliance with a terminal such as a head-mounted device, a headphone, or a left-right separated in-ear headphone. In the case of such a structure, the direction acquisition unit 10 and the reproduction unit 40 can be set in the terminal, and the functions of the direction acquisition unit 10 and the sound and image translation unit 20 are performed by one of the information processing device or the terminal. In addition, information can be transmitted between the information processing device and the terminal using, for example, Bluetooth (registered trademark), HDMI (registered trademark), WiFi (registered trademark), USB (Universal Serial Bus) or other wired or wireless information transmission methods. In such a case, the functions of the information processing device can also be performed by a server on an intranet or the Internet.

[0269] Furthermore, in the first and second embodiments described above, the sound reproducing device 1 is described as including the output unit 30 and the reproducing unit 40. However, a configuration not including the output unit 30 and the reproducing unit 40 is also possible. Figure 5 An example of the structure of such a sound generating device 2b that generates only a sound signal is described. In the sound generating device 2b, for example, the data of the generated sound signal can be stored in the recording medium M.

[0270] In addition, the sound generating device 2b of such another embodiment can be used in the following devices: content reproduction devices such as PCs, smartphones, game devices, and media players; and devices used in various devices such as VR, AR, MR, video phones, video conference systems, teleconference systems, game devices, and other home appliances. In other words, the sound generating device 2b can be applied to all devices that can obtain the direction of the sound source S in the virtual space, such as devices equipped with a television or a display, video phones, video conferences, and telepresence through a display.

[0271] In addition, the sound signal processing program of this embodiment can also be executed by these devices. In addition, when producing and distributing content, these sound signal processing programs can also be executed on a PC, server, etc. of a production company or distribution source. In addition, the sound signal processing program can also be executed in the sound reproduction device 1 of the above-mentioned embodiment.

[0272] That is, through the processing of the above-mentioned sound generating device 2, 2b and / or the sound signal processing program, it is possible to reproduce movies, games, VR, AR, MR, etc. with a higher sense of presence and authenticity through headphones and / or headphones of HMD. In addition, the sense of presence can be improved even in remote conferences, etc. In addition, applications to movie theaters, field games, 3D sound field capture, transmission, and reproduction systems, as well as applications to AR, VR applications, etc. have also become possible.

[0273] In the first and second embodiments described above, an example of adding direction information to the sound signal of the sound source S is described. In contrast, in a situation where a speaker and a listener are having a conversation at any time such as the above-mentioned teleconference, it is also possible not to add direction information to the sound signal of the sound source S. That is, the direction of the speaker (the current listener) can be estimated using the sound signal of the speaker when the current listener is the speaker, and the direction can be used as the direction of the listener observed from the current speaker.

[0274] In such a case, the direction acquisition unit 10 calculates the arrival direction of the signal of the L (left) channel (hereinafter referred to as "L signal") and the signal of the R (right) channel (hereinafter referred to as "R signal") of the sound signal observed by the listener U of the sound signal. At this time, the direction acquisition unit 10 can obtain the intensity ratio of the L channel and the R channel. Based on this intensity ratio, the arrival direction of the signal of each frequency component can also be estimated.

[0275] Alternatively, the direction acquisition unit 10 may estimate the direction of arrival of the sound signal based on the relationship between the ITD (Interaural Time Difference) and the direction of arrival of the signal of each frequency in the HRTF (Head-Related Transfer Function). The relationship between the ITD and the direction of arrival may also be referred to by the direction acquisition unit 10 by the content stored in the storage unit as a database.

[0276] Alternatively, it is also possible to perform facial recognition based on facial image data of a speaker or listener in a content or video conference to estimate the direction of the speaker or listener. In other words, even in a structure without head tracking, the direction can be estimated. Similarly, it is also possible to grasp the position of the speaker and listener in the space.

[0277] By configuring in this way, various flexible structures can be handled. In addition, in applications such as VR and social VR, the position of the sound source is known in advance, so the direction of the sound source S can be obtained based on the positional relationship between the sound source S and the listener U, without estimating the direction of the sound source.

[0278] Next, the sound generating device will be further described according to embodiments based on the drawings, but the following specific examples do not limit the sound generating device.

[0279] [Example 1]

[0280] (SNR comparison using my own HRTF)

[0281] In this experiment, HRIRs of the HRTF of the subject (listener) himself (hereinafter referred to as "original") were actually produced at intervals of 15°, and HRIRs after converting them into HRIRs were produced. In addition, for the original HRIR, representative points were set, and time shifting was performed on the entire circumference of the horizontal plane (left-right direction) using the time shift value based on the cross-correlation of the above-mentioned embodiment, and sound image panning using two representative points was performed using the gain value calculated by the above-mentioned vector calculation (hereinafter referred to as "sound image panning of this embodiment").

[0282] Specifically, first, a comparison experiment was conducted between the result obtained by convolving the sound source S with the original HRIR (hereinafter referred to as the "true value") and the sum obtained by adding the results obtained by convolving each HRIR of the two representative points with the HRIR of the present embodiment after sound image panning (hereinafter referred to as the "approximate value"). In addition, in fact, in order to simplify the processing sequence, a signal equivalent to the above-mentioned "approximate value" is generated by adding the results obtained by applying each gain to the results of time-shifting the HRIRs of the two representative points, simulating the HRIR in the direction of the sound source (hereinafter referred to as the "synthesized HRIR"), and convolving it with the sound source signal.

[0283] In addition, as a comparative example, a conventional gain based on the conventional sine law without time shift is used. In the sine law of this comparative example, when the angle from the front to the sound source S is θ and the angle from the front to the representative point R is θ0, the left gain A used for multiplying the sound source signal convolved with the HRIR using two representative points is s and right gain B s Calculated as (A s -B s ) / (A s +B s )=sinθ / sinθ0.

[0284] The representative points used in this embodiment are set in the representative point directions of (1) a range angle of 90° (45°, 135°, 225°, 315°), (2) a range angle of 90° (0°, 90°, 180°, 270°), and (3) a range angle of 60° (30°, 90°, 150°, 210°, 270°, 330°). These groups of representative points are respectively referred to as (1) 4 directions_oblique, (2) 4 directions_vertical and horizontal, and (3) 6 directions. For these embodiments and comparative examples, the difference between the output signal obtained by convolution with the HRIR of each sound source direction and the "approximate value" is calculated as the SNR.

[0285] Reference Figure 6 to Figure 11 The results are described below. In each of the graphs, the horizontal axis represents the angle, and the vertical axis represents the SNR (dB, decibel).

[0286] Figure 6 It shows the results of SNR comparison (4 directions_oblique, right ear).

[0287] Figure 7 It shows the results of SNR comparison (4 directions_oblique, left ear).

[0288] Figure 8 It shows the result of SNR comparison (4 directions_vertical and horizontal, right ear).

[0289] Fig. 9 It shows the result of SNR comparison (4 directions_vertical and horizontal, left ear).

[0290] Fig.10 Shows the results of SNR comparison (6 directions, right ear).

[0291] Fig.11 Shows the results of SNR comparison (6 directions, left ear).

[0292] In any case, the SNR is 5 to 10 dB higher than that of the comparative example. Thus, by using the sound image panning of this embodiment, the SNR can be improved compared to the conventional method.

[0293] (Positioning experiment based on subjective evaluation)

[0294] Next, an experiment (localization experiment) was conducted to measure subjective localization by subjects using the true value after convolution of the original HRIR and the approximate value based on the sound image panning of this embodiment. The conditions of this localization experiment are shown in Table 1 below.

[0295] [Table 1]

[0296]

[0297] The prompt sound pressure is measured by wearing headphones on the dummy's head and using a measuring amplifier. Figure 12 to Figure 15 In each graph, the horizontal axis represents the direction of the sound source prompted, and the vertical axis represents the direction of the listener's answer. That is, if the direction matches the 45° line, it means that the listener correctly identified the direction of the sound source prompted. The same part in the two tests is represented by a larger circle, and the different parts in the two tests are represented by a smaller circle.

[0298] Fig.12 represents the results of a localization experiment that indicates the subjective localization of a sound source S using a true value. Fig.12 Although the true value results are partially deviated from the oblique direction, the sound source direction answered by the listener is generally correct, that is, along the line of about 45° on the graph.

[0299] Fig.13 The results of the positioning experiment using the representative points in the four directions (1) mentioned above are shown.

[0300] Fig.14 The results of the positioning experiment using representative points in the four directions (2) mentioned above - vertical and horizontal - are shown.

[0301] Fig.15 The results of the positioning experiment using representative points in the six directions described in (3) above are shown.

[0302] exist Figure 13 to Figure 15 , (a) is an example of using a gain based on the sine law as a comparative example, and (b) is an example of an approximate value caused by sound image panning of a representative point of this embodiment.

[0303] As a result, in the comparative example in which the sine law sound image panning was used, the degree to which the sound source direction could be identified in six directions was higher than the degree to which the sound source direction could be identified in four directions, but the listener could not identify the sound source direction correctly.

[0304] In contrast, in the approximation value of the representative point of the present embodiment caused by the sound image translation, it is very close to the true value and is basically along the 45° line. In the approximation value of the present embodiment, it can be seen that even in the four directions-diagonal directions, the angles are almost along the 45° line. That is, in the approximation value of the present embodiment, the number of representative points can be reduced, and the representative points in the four directions are sufficient for the listener to identify the direction of the sound source.

[0305] That is, in the sound image panning of the present embodiment, in the case of using full white noise, the listener can fully recognize the direction of the sound source compared to the original HRIR.

[0306] (Subjective quality evaluation using the MUSHRA method)

[0307] Next, the speech sound source is used to evaluate the degree of change in the timbre of the sound source S. Specifically, the MUSHRA (MUltiple Stimuli with Hidden Reference and Anchor) method is used to evaluate whether the approximation based on the sound image panning of this embodiment has changed compared to the original HRIR convolved with the speech sound source. The MUSHRA method is a method for measuring the subjective quality of audio defined in ITU-R BS.1534.

[0308] Here, similarly to the other experiments described above, the original HRIR of the comparative example, and the synthetic HRIR of the voice image translation of the present embodiment are compared with the JVS (Japanese Versatile Speech) corpus (<URL=“https: / / sites.google.com / site / shinnosuketakamichi / research-topics / jvs_corpus”> ) convolution to generate and evaluate (true value) and (approximate value). The experimental conditions in the MUSHRA method are shown in Table 2 below.

[0309] [Table 2]

[0310]

[0311] (※) 12 types: 15°, 60°, 75°, 105°, 120°, 165°, 195°, 240°, 255°, 285°, 300°, 345°

[0312] In this experiment, after listening to the result (true value) obtained by convolving the speech sound with the original HRIR, excluding the angle where the sound source is located, the evaluation of each embodiment and comparative example including (true value) is randomly listened to and a blind evaluation is performed.

[0313] Fig.16 The experimental results of subjective quality evaluation using the MUSHRA method (one type of male voice) are shown. In each graph, A is the original (true value), B is 4 directions_diagonal (comparative example), C is 4 directions_vertical and horizontal (comparative example), D is 6 directions (comparative example), E is 4 directions_diagonal (implementation), F is 6 directions_vertical and horizontal (implementation), and G is 6 directions (implementation). In the graphs, the vertical axis represents the evaluation score, the horizontal bar marked with an x ​​represents the average value of the evaluation score, and the height of the bar represents the 95% confidence interval.

[0314] As a result, the ranking is original (true value), this embodiment, and comparative example. That is, it can be seen that the sound image panning of this embodiment is close to the evaluation score of the original HRIR, and the evaluation score is higher than that of the conventional sine law.

[0315] (Comparison of SNR of HRIR using FABIAN)

[0316] The original HRIR mentioned above is at 15° intervals. Therefore, in order to objectively evaluate within a narrower angle range, FABIAN (<URL=“https: / / depositonce.tu-berlin.de / handle / 11303 / 6153”> This FABIAN includes data at 2° intervals. Since FABIAN is not the subject's own HRIR, an objective evaluation of SNR was performed and the results were confirmed for the sound image panning performed in this embodiment.

[0317] The representative points used in this embodiment are the same as those used in the original case. That is, in the representative point direction, (1) a range angle of 90° (45°, 135°, 225°, 315°), (2) a range angle of 90° (0°, 90°, 180°, 270°), and (3) a range angle of 60° (30°, 90°, 150°, 210°, 270°, 330°) are set. These groups of representative points are respectively referred to as (1) 4 directions_oblique, (2) 4 directions_vertical and horizontal, and (3) 6 directions.

[0318] In the sound image panning of this embodiment using this FABIAN, time shifting based on cross-correlation is also performed, and the gain obtained by vector calculation is used. Figures 17 to 23 The results are explained below. In each figure, the horizontal axis represents the angle and the vertical axis represents the SNR (dB). Figure 17 to Figure 19 In the figure, (a) shows the result of the left ear, and (b) shows the result of the right ear.

[0319] Fig.17 It shows the result of (1) SNR (4 directions_oblique).

[0320] Fig.18 It shows the result of (2) SNR (4 directions_vertical and horizontal).

[0321] Fig.19 The results of (3) SNR (6 directions) are shown.

[0322] Fig. 20 The following table shows the results of the SNR comparison (right ear) of the three types (1) to (3).

[0323] Fig.21 The following table shows the results of the SNR comparison (left ear) of the three types (1) to (3).

[0324] Fig. 22 The results of SNR comparison (right ear) of (1) and (2) in only four directions are shown.

[0325] Fig.23The results of SNR comparison (left ear) of (1) and (2) in only four directions are shown.

[0326] according to Figure 17 to Figure 19 As a result, in the case of 4 directions, the SNR is about 10dB at a good angle and about 6dB at a bad angle. In addition, the result of (2) 4 directions_vertical and horizontal is better than the result of (1) 4 directions_oblique. In other words, when it is 4 directions_vertical and horizontal, the SNR exceeds 20dB at a good angle and is about 10dB at a bad angle. In addition, since FABIAN is a data with a scale of 2°, it is easy to see the behavior of each angle.

[0327] Figure 20-21 The method is to overlap all 4 directions and all 6 directions to determine which one is the best. As a conclusion, 4 directions are sufficient.

[0328] exist Figure 22-23 In the above example, only four directions are overlapped to determine which is better, vertical, horizontal or diagonal. As a conclusion, it can be seen from the graph that (2) 4 directions_vertical and horizontal is better than (1) 4 directions_diagonal, and using 4 vertical and horizontal positions is better than using 4 diagonal positions.

[0329] (The effect of decimal shift)

[0330] In the above FABIAN verification, there is a large difference in SNR between adjacent angles, and a comb-shaped shape appears. Therefore, the time shift amount used in the sound image panning of this embodiment is confirmed. Figure 24 to Figure 29 The time shift when the cross-correlation of each angle is the maximum is shown in . The horizontal axis shows the angle, and the vertical axis shows the time shift (number of samples). "Endpoint 1" represents the representative point R-1, and "endpoint 2" represents the representative point R-2.

[0331] Fig.24 It represents the calculation result of the time shift (4 directions_diagonal, right ear).

[0332] Fig.25 Indicates the calculation result of the time shift (4 directions_diagonal, left ear).

[0333] Fig.26 It indicates the calculation result of the time shift (4 directions_vertical and horizontal, right ear).

[0334] Fig. 27 It indicates the calculation result of the time shift (4 directions_vertical and horizontal, left ear).

[0335] Fig.28 Indicates the calculation result of the time shift amount (6 directions, right ear).

[0336] Fig.29 Indicates the calculation result of the time shift amount (6 directions, left ear).

[0337] The curves are all scaled at 2°, but there are also several points where the time shift is equal. Here, in the above embodiment, the time shift is performed to maximize the cross-correlation, but the shift is only an integer value. Therefore, it is considered that there is a position where the amount of the intended shift and the actual shift amount are different. For example, it is considered that there is a position where the intended shift amount is 0.6 samples but the actual shift amount becomes 1 sample.

[0338] That is, since the sampling frequency of the sound source S is only time-shifted by an integer value, even when the value of the most appropriate shifted sample is a decimal, it becomes an integer. Therefore, the inventors of the present invention thought that it is possible to perform a substantial decimal shift by oversampling, reduce the deviation of the shift amount, and expect an improvement in SNR, and verified it. In other words, they thought of maximizing the cross-correlation by shifting by 0.5 samples, 0.25 samples, etc., and verified it.

[0339] Here, 4-fold oversampling was performed, and the SNR was compared with that in the case of integer shifting (Example). Specifically, it was verified whether the cross-correlation could be maximized by 4-fold oversampling from 48 kHz sampling used in FABIAN's HRIR to 192 kHz.

[0340] This is because the spatial length of one sample in 48kHz sampling is about 0.7cm, and when oversampling is performed by 4 times, the spatial length of each sample is about 0.18cm. Therefore, if the size of a person's face and ears is taken into consideration, this level of resolution is sufficient.

[0341] The effect of such oversampling-based fractional shifting was verified by FABIAN's HRIR. Figure 30 to Figure 35 The graphs show the results of comparing the SNR between integer shift and fractional shift. The horizontal axis of the graphs represents the angle, and the vertical axis represents the SNR (dB, decibel).

[0342] Fig.30 Shows the results of SNR comparison (4 directions, oblique, right ear).

[0343] Fig.31 Shows the results of SNR comparison (4 directions, oblique, left ear).

[0344] Fig.32 Shows the results of SNR comparison (4 directions, vertical and horizontal, right ear).

[0345] Fig.33 Shows the results of SNR comparison (4 directions, vertical and horizontal, left ear).

[0346] Fig.34Shows the results of SNR comparison (6 directions, right ear).

[0347] Fig.35 Shows the results of SNR comparison (6 directions, left ear).

[0348] By performing fractional shifting, the SNR variation of the comb shape caused by the angle is suppressed, thereby further improving the SNR.

[0349] (Study on the amount of computation)

[0350] Next, since the amount of calculation increases when oversampling is performed for fractional shifting, the increase in the amount of calculation caused by this is studied. Specifically, by estimating the amount of calculation, the extent of the increase in the amount of calculation caused by oversampling is estimated and confirmed.

[0351] The amount of computation was estimated under the following conditions.

[0352] The number of sound source objects (sound sources S) within the range angle: M

[0353] HRIR tap number: L

[0354] Number of oversampling filters used for fractional shifting: N

[0355] (In the case of N times oversampling)

[0356] The time shift value for the number of points (including decimals: 3.25 points, etc.) to be shifted by M-fold oversampling is calculated in advance according to the direction of the sound source S (sound source direction) of each HRIR.

[0357] The sound source S is time-shifted by the time-shift value.

[0358] As a comparative example, for each sound source S, the amount of calculation when directly convolving the HRIR in the direction of the sound source S (sound source direction) and when using the sound image panning of this embodiment are as follows (α), (β), and (γ).

[0359] (α) When convolution is performed separately without panning

[0360] The amount of calculation required for each sample (number of product sums): ML

[0361] (β) Oversampling and panning with fractional shifts

[0362] Calculation of 1 oversampling point: 2N

[0363] Oversample all sound sources S: 2MN

[0364] Calculate the value of the representative point: 2M+2(M-1)

[0365] ≒(Apply gain to two representative points) + (Generate sum signal to two representative points)

[0366] Convolution: 2L

[0367] The amount of calculation required for each sample (number of product sums): 2MN+2M+2(M-1)+2L

[0368] (γ) No oversampling (reference)

[0369] The amount of calculation required for each sample (number of product sums): 2M+2(M-1)+2L

[0370] Here, a specific example of the comparison of the amount of calculation using the above-mentioned methods (α) and (β) is described. In either case, the order N of the oversampling filter is set to 16.

[0371] i. The number of sound source objects: M = 3, the number of HRIR taps: L = 256

[0372] The number of operations in (α): 3×256=768

[0373] The number of operations in (β): 2×3×16+2×3+2(3-1)+2×256=618

[0374] ii. Number of sound source objects: M = 4, number of HRIR taps: L = 256

[0375] The number of operations in (α): 4×256=1024

[0376] The number of operations in (β): 2×4×16+2×4+2(4-1)+2×256=654

[0377] As a result, the sum of products is reduced to 65-80%.

[0378] (Example of waveform)

[0379] Fig.36 An example of comparing the waveform of the synthetic HRIR based on the panning of the present embodiment described above with the waveform of the subject's own (original) HRIR is shown. Here, a representative example of comparing the waveforms (4 directions_diagonal) at the rear (135° to 225°) is shown. The upper figure is the waveform of the synthetic HRIR based on the panning of the present embodiment, and the lower figure is the waveform of the original HRIR.

[0380] Fig.37A representative example is shown comparing the waveform of the synthesized HRIR based on the sound image panning of the present embodiment described above with the waveform of the HRIR of FABIAN. Here, regarding the waveform of (4 directions_oblique, right ear), the upper figure shows the waveform of the synthesized HRIR based on the sound image panning of the present embodiment, and the lower figure shows the waveform of the HRIR of FABIAN.

[0381] In any case, it can be seen that the waveforms are very similar. The same is true for other waveforms. That is, the sound image panning of this embodiment can be approximated with good accuracy. That is, by panning the sound image of a specific representative direction and synthesizing the sound source, the HRIR of the sound source direction can be equivalently generated by the HRIR of the representative direction.

[0382] [Example 2]

[0383] An HRIR is generated by applying weighted filtering to the impulse response of the LPF with a cutoff frequency of 3000 Hz and an attenuation frequency of 8 dB / Oct as shown in the second embodiment and calculating the cross-correlation, and the original HRIR is compared with the case where no weighted filtering is applied.

[0384] Specifically, Fig.38 The figure shows the result of measuring the envelope of the input waveform of the left ear when a 1 kHz sine wave travels around the head from the front to the left in 8 seconds. Fig.38 (a) represents the result using the original HRIR, (b) represents a comparative example, in which the HRIR in six directions is shifted by one integer layer without weighted filtering and the result is measured, and (c) represents the result in this embodiment, in which the HRIR in six directions is shifted by one integer layer with weighted filtering and the result is measured.

[0385] As a result, compared with the comparative example, by applying weighted filtering, the moving sound source is smoothly transitioned and close to the original HRIR is achieved.

[0386] In addition, the configuration and operation of the above-described embodiment are merely examples, and can of course be appropriately modified and executed without departing from the gist of the present disclosure.

[0387] Industrial Applicability

[0388] The sound generating device disclosed in the present invention can reduce the amount of calculations when generating stereo sound and reduce the load, and can be used in industry.

[0389] Description of Reference Numerals

[0390] 1Sound reproduction device

[0391] 2.2b Sound generating device

[0392] 10. Direction Acquisition Department

[0393] 20. Audio and video translation

[0394] 30 output unit (sound output unit)

[0395] 40 Recapitulation

[0396] 200HRIR Table

[0397] M Recording Media

[0398] R-1, R-2, R-3, R-4 represent points

[0399] S, S-1, S-2, S-3, S-4, Sn sound source

[0400] UListener

Claims

1. A sound generating device, characterized in that: have: a direction acquisition unit that acquires a sound source direction of a sound source; and The sound image panning unit pans the sound from a specific representative direction by time shifting and gain adjusting the sound source based on the sound source direction acquired by the direction acquisition unit, thereby expressing the sound source.

2. The sound generating device according to claim 1, characterized in that: There are multiple sources of said sound, The representative directions are directions for each representative point, and the number of the representative directions is less than the number of the sound sources. The sound image translation unit synthesizes the sound images of the plurality of sound sources using the plurality of sounds representing the directions.

3. The sound generating device according to claim 2, characterized in that: The sound image panning unit performs a time shift calculated to maximize the cross-correlation between the head impulse response in the sound source direction and the head impulse response in the representative direction, or a time shift in which a negative sign is added to the time shift, on the sound source.

4. The sound generating device according to claim 3, characterized in that: The time shift and / or gain is obtained by calculating the cross-correlation after applying weighted filtering on the frequency axis.

5. The sound generating device according to claim 3, characterized in that: The sound image shifting unit applies a gain set for each of the sound sources and each of the representative directions to the time-shifted sound sources at each of the plurality of representative points.

6. The sound generating device according to claim 5, characterized in that: When synthesizing the HRIR vector of the sound source direction by summing the HRIR vectors of the representative directions, the sound image panning unit uses a gain calculated so that an error signal vector between the synthesized HRIR vector and the HRIR vector of the sound source direction is orthogonal to the HRIR vector of the representative direction.

7. The sound generating device according to claim 5, characterized in that: The sound image panning unit uses a gain calculated so as to minimize the energy or L2 norm of an error signal vector between the synthesized HRIR vector and the HRIR vector in the sound source direction.

8. The sound generating device according to claim 7, characterized in that: The error signal vector uses a result obtained by applying weighted filtering on the frequency axis.

9. The sound generating device according to claim 5, characterized in that: The sound image panning unit uses the corrected gain so that the energy balance of the head impulse responses of the left ear and the right ear from the position of the sound source is maintained also in the head impulse response substantially synthesized from the head impulse responses from the plurality of representative points through sound image panning.

10. The sound generating device according to claim 5, characterized in that: The sound image translation unit performs the time shift on the sound source, and processes the signal obtained by applying the gain as a representative point signal existing at the position of the representative point, and convolves the head impulse response at the position of the representative point with the sum signal of the representative point signals of the number of the sound sources to generate a signal at the listener's ear.

11. The sound generating device according to claim 3, characterized in that: The time shift also allows a shift of the decimal point of the samples.

12. The sound generating device according to claim 3, characterized in that: The tendency of high-frequency region attenuation is compensated by reproducing high-frequency region enhancement filtering.

13. The sound generating device according to claim 1, characterized in that: The sound source is one of a sound signal of the content and a sound signal of a participant in the remote call, The direction acquisition unit acquires the direction of the sound source as viewed from a listener.

14. A sound reproduction device, characterized in that: have: The sound generating device according to any one of claims 1 to 13; and The sound output unit outputs the sound signal generated by the sound generating device.

15. A sound generation method, which is a sound generation method performed by a sound generation device, characterized in that: Get the sound source direction of the sound source, Based on the acquired sound source direction, the sound source is represented by performing a sound image panning of the sound from a specific representative direction through time shift and gain adjustment of the sound source.

16. A sound signal processing program, the sound signal processing program being executed by a sound generating device, characterized in that: The sound signal processing program causes the sound generating device to: Get the sound source direction of the sound source, Based on the acquired sound source direction, the sound source is represented by performing a sound image panning of the sound from a specific representative direction through time shift and gain adjustment of the sound source.

Images