Audio processing method, audio processing device, and audio processing program
The sound processing method localizes sound images outside the user's head and moves them along the periphery or around the user, using synthetic binaural room impulse responses to enhance three-dimensional sound perception.
Patent Information
- Application Number
- JP2024140853
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2024-08-22
- Publication Date
- 2024-11-08
AI Technical Summary
Conventional sound processing devices struggle to effectively convey the effect of extra-head localization of a sound image, making it difficult for users to perceive three-dimensional sound clearly, especially when using headphones.
A sound processing method that localizes a sound image outside the user's head and moves it along the periphery or around the user, utilizing synthetic binaural room impulse responses to enhance the out-of-head localization effect.
The method clearly communicates the effect of extra-head localization, improving the perception of three-dimensional sound by enhancing the difference in sound pressure and phase characteristics between ears.
Smart Images

Figure 2024159912000001_ABST
Abstract
Description
[Technical Field]
[0001] The present disclosure relates to a technology for localizing a sound image outside a user's head. [Background technology]
[0002] BACKGROUND ART Conventionally, there is known a sound processing device that allows a user to perceive a reproduced sound as a sound arriving from a predetermined direction in a three-dimensional sound field (see, for example, Patent Document 1). [Prior art documents] [Patent documents]
[0003] [Patent Document 1] International Publication No. 2023 / 106070 Summary of the Invention [Problem to be solved by the invention]
[0004] However, with the above-mentioned conventional technology, it is difficult to convey the effect of out-of-head localization of sound images to the user in an easily understandable manner, and further improvement is needed.
[0005] The present disclosure has been made to solve the above problems, and aims to provide a technology that can clearly convey to a user the effect of out-of-head localization of a sound image. [Means for solving the problem]
[0006] An acoustic processing method according to one aspect of the present disclosure is an acoustic processing method executed by a computer, and includes receiving an instruction to start acoustic processing that causes a user to perceive a reproduced sound as a sound arriving from a predetermined direction in a three-dimensional sound field, localizing a sound image of the starting sound at a starting position outside the user's head, and moving the localized sound image so that it passes to the side of the user. [Effects of the Invention]
[0007] According to the present disclosure, the effect of out-of-head localization of a sound image can be conveyed to a user in an easily understandable manner. [Brief explanation of the drawings]
[0008] [Figure 1] 1 is a schematic diagram showing a first example of use of the sound reproducing device according to the present embodiment. [Figure 2] FIG. 10 is a schematic diagram showing a second example of use of the sound reproducing device according to the present embodiment. [Figure 3] 1 is a block diagram showing a functional configuration of a sound reproducing device according to an embodiment of the present invention. [Figure 4] FIG. 2 is a block diagram showing a more detailed functional configuration of the sound processing device according to the present embodiment. [Figure 5] FIG. 2 is a schematic diagram for explaining a first sound image localization process for moving a sound image localized outside the head in the present embodiment. [Figure 6] FIG. 10 is a schematic diagram for explaining a second sound image localization process for moving a sound image localized outside the head in this embodiment. [Figure 7] FIG. 10 is a schematic diagram for explaining a third sound image localization process for moving a sound image localized outside the head in this embodiment. [Figure 8] FIG. 10 is a schematic diagram for explaining a fourth sound image localization process for moving a sound image localized outside the head in this embodiment. [Figure 9] 10A and 10B are diagrams for explaining volume attenuation according to the present embodiment. [Figure 10] 3A and 3B are diagrams for explaining the arrival direction of a reproduced sound output by a sound processing device according to the present embodiment. [Figure 11] FIG. 2 is a block diagram showing a configuration of a first signal processing unit according to the present embodiment. [Figure 12] FIG. 4 is a block diagram showing a configuration of a second signal processing unit according to the present embodiment. [Figure 13] FIG. 2 is a diagram for explaining a method for creating a first synthesized binaural room impulse response in the present embodiment. [Figure 14]FIG. 2 is a block diagram showing a configuration of a first creation unit according to the present embodiment. [Figure 15] FIG. 3 is a block diagram showing a configuration of a second creation unit according to the present embodiment. [Figure 16] 4 is a flowchart showing the operation of the sound processing device according to the present embodiment. [Figure 17] 10 is a flowchart illustrating a first output sound signal generation process by a first processing unit according to the present embodiment. [Figure 18] 10 is a flowchart illustrating a second output sound signal generation process by a second processing unit according to the present embodiment. [Figure 19] 10 is a flowchart illustrating a first synthesized binaural room impulse response creation process performed by a first creation unit according to the embodiment of the present disclosure. [Figure 20] 3A to 3C are diagrams showing an example of a first binaural room impulse response and a first head-based impulse response in the present embodiment. [Figure 21] FIG. 2 is a diagram showing an example of a first synthesized binaural room impulse response in the present embodiment. [Figure 22] FIG. 10 is a diagram for explaining a process for aligning the maximum amplitude position of the first binaural room impulse response and the maximum amplitude position of the first head-based impulse response in the present embodiment. [Figure 23] FIG. 4 is a diagram for explaining an example of an appropriate first angle according to the present embodiment. [Figure 24] FIG. 4 is a diagram for explaining an example of an appropriate first delay time according to the present embodiment. [Figure 25] 5A and 5B are diagrams illustrating an example of an appropriate first volume attenuation amount according to the present embodiment. [Figure 26] FIG. 10 is a block diagram showing a configuration of a first creation unit in a first modified example of the present embodiment. [Figure 27] FIG. 10 is a block diagram showing the configuration of a second creation unit in a first modified example of the present embodiment. [Figure 28]10 is a flowchart illustrating a first synthesized binaural room impulse response creation process by a first creation unit according to a first modification of the embodiment of the present disclosure. [Figure 29] FIG. 10 is a diagram showing an example of a first binaural room impulse response in Modification 1 of the present embodiment. [Figure 30] FIG. 10 is a diagram illustrating a process of replacing all amplitudes of direct sound components of a first binaural room impulse response with 0 in the first modification of the present embodiment. [Figure 31] FIG. 10 is a diagram illustrating a process of setting the amplitude of the first binaural room impulse response at time 0 to 1 in the first modification of the present embodiment. [Figure 32] FIG. 10 is a diagram showing an example of a first synthesized binaural room impulse response in Modification 1 of the present embodiment. [Figure 33] FIG. 10 is a block diagram showing the functional configuration of a sound reproducing device according to a second modification of the present embodiment. [Figure 34] FIG. 10 is a block diagram showing a detailed functional configuration of a sound processing device according to a second modification of the present embodiment. [Figure 35] FIG. 10 is a diagram for explaining the direction in which a sound output by a sound processing device arrives, according to a second modification of the present embodiment. [Figure 36] 10 is a flowchart showing the operation of a sound processing device according to a second modification of the present embodiment. DETAILED DESCRIPTION OF THE INVENTION
[0009] (Findings that formed the basis of this disclosure) Conventionally, there is known a technology for sound reproduction that allows a user to perceive three-dimensional sound by controlling the position of a sound image, which is a sound source object perceived by the user, in a virtual three-dimensional space (hereinafter sometimes referred to as a three-dimensional sound field). By localizing a sound image at a predetermined position in the virtual three-dimensional space, the user can perceive the sound as if it were arriving from a direction parallel to a line connecting the predetermined position and the user (i.e., a predetermined direction). To localize a sound image at a predetermined position in the virtual three-dimensional space in this way, for example, calculation processing is required to generate a difference in sound arrival time between the two ears and a difference in sound level (or sound pressure) between the two ears, etc., for the collected sound so that the sound is perceived as three-dimensional sound.
[0010] In recent years, so-called online conference systems have become increasingly popular, enabling communication with a communication partner by bidirectionally transmitting and receiving video and audio over a communication line. Head-mounted audio playback devices such as headphones are often used in these online conference systems. When listening to sound through headphones, as in the case of the above-mentioned online conference systems, it is difficult to expand the sound into a three-dimensional sound field and allow the user to perceive it. For example, it is known that a sufficient out-of-head sensation cannot be achieved by simply defining the direction of a display device displaying the communication partner as the sound arrival direction and convolving a head-related transfer function that perceives the sound as arriving from this direction. That is, because the sound image is localized within the user's head, a sense of incongruity arises between the image of the communication partner on the display device and the sound localized within the user's head. Continuing to listen while experiencing this sense of incongruity can result in unnecessary fatigue. A similar issue can also arise when listening to content using three-dimensional visual spaces, such as virtual reality (VR) and augmented reality (AR), through audio playback devices such as headphones.
[0011] Therefore, there are known techniques that can expand sound into a three-dimensional sound field even when using headphones. For example, one technique involves simulating how reflected sounds occur in a simulated room, artificially creating and synthesizing these reflected sounds, and having the user listen to them. The sound, including the synthesized reflected sounds, allows the user to perceive the original sound as if it were coming from a specific direction in the simulated room.
[0012] In the sound processing devices of the above-mentioned conventional technology, the sound image of the reproduced sound is mainly localized in the direction directly in front of the user. The effect of out-of-head sound localization varies depending on the user, and some users find the effect difficult to perceive. In particular, there is a tendency for the sound image to be difficult to localize when it is directly in front of the user. For this reason, with the above-mentioned conventional technology, it has been difficult to convey the effect of out-of-head sound localization to the user in an easy-to-understand manner.
[0013] In order to solve the above problems, the following techniques are disclosed.
[0014] (1) An acoustic processing method according to one aspect of the present disclosure is an acoustic processing method executed by a computer, and includes receiving an instruction to start acoustic processing that causes a user to perceive a reproduced sound as a sound arriving from a predetermined direction in a three-dimensional sound field, localizing a sound image of the starting sound at a starting position outside the user's head, and moving the localized sound image so that it passes to the side of the user.
[0015] When the sound image is to the side of the user, the difference in sound pressure characteristics and phase characteristics between the two ears becomes greater than when the sound image is in front of the user, making it easier to localize the sound image. With the above configuration, the sound image of the start sound passes to the side of the user, so the effect of out-of-head localization of the sound image can be conveyed to the user in an easily understandable manner.
[0016] (2) In the acoustic processing method described in (1) above, the start position may be located to the side of the user, and the movement may include positioning the sound image at the start position and moving the sound image around the user from the start position toward a playback position located in the specified direction to which the playback sound arrives.
[0017] According to this configuration, the sound image of the starting sound is moved from the side of the user toward the playback position along the periphery of the user, thereby enhancing the effect of localizing the sound image.
[0018] (3) In the acoustic processing method described in (1) above, the start position may be located to the side of the user, and the movement may include positioning the sound image at the start position and moving the sound image from the start position around the user, passing behind the user, toward a playback position located in the specified direction to which the playback sound arrives.
[0019] With this configuration, the sound image of the starting sound moves from the side of the user, past the rear of the user, and around the user toward the playback position, thereby enhancing the effect of localizing the sound image.
[0020] (4) In the acoustic processing method described in (1) above, the start position may be located to the side of the user, and the movement may include positioning the sound image at the start position, moving the sound image from the start position around the user toward a playback position located in the specified direction to which the playback sound arrives, and further moving the sound image around the user from the playback position.
[0021] With this configuration, the sound image of the starting sound moves from the side of the user toward the playback position along the user's periphery, and then moves around the user from the playback position, thereby enhancing the effect of localizing the sound image.
[0022] (5) In the acoustic processing method described in (1) above, the start position may be a playback position located in the specified direction to which the playback sound arrives, and the movement may include positioning the sound image at the start position and moving the sound image around the user from the start position.
[0023] According to this configuration, the sound image of the starting sound is made to circle around the user from the playback position, thereby enhancing the effect of localizing the sound image.
[0024] (6) The sound processing method according to any one of (1) to (5) above may further include playing the start sound when the start instruction is accepted.
[0025] According to this configuration, when a start instruction is accepted, a start sound different from the playback sound is played, so that the start sound can notify the user that acoustic processing for localizing a sound image outside the head is about to begin.
[0026] (7) In the acoustic processing method described in any one of (1) to (5) above, the movement may include, when the start instruction is accepted, localizing the sound image of the playback sound being played when the start instruction is accepted at the start position as the sound image of the start sound.
[0027] According to this configuration, when a start instruction is accepted, the sound image of the playback sound being played when the start instruction is accepted is localized at the start position as the sound image of the start sound, so that the user can be notified by the playback sound that acoustic processing for localizing the sound image outside the head is starting.
[0028] (8) In the acoustic processing method described in any one of (1) to (7) above, the movement may include generating a first output sound signal by convolving sound information including the start sound with a first head-related transfer function for localizing the sound as a sound arriving from the direction of the sound image; generating a second output sound signal by convolving the sound information with a second head-related transfer function for localizing the sound as a sound arriving from a first direction having a first angle with the direction of the sound image that is greater than 0 degrees and less than 360 degrees, and having a first delay time greater than 0 and a first volume attenuation greater than 0 with respect to the start sound perceived by the first output sound signal; and outputting an output sound signal obtained by combining the generated first output sound signal and the second output sound signal.
[0029] According to this configuration, the second output sound signal arrives from the first direction and is localized as a sound having a first delay time and a first volume attenuation amount, so that the user perceives the reflected sound as the initiation sound reflected by the pseudo-reflection wall. Therefore, the reflected sound having the first delay time and the first volume attenuation amount is perceived together with the initiation sound as a direct sound, improving the out-of-head feeling at the position where the sound image of the direct sound is localized.
[0030] (9) In the sound processing method according to any one of (1) to (7) above, the movement includes generating a first output sound signal by convolving sound information including the start sound with a first synthesized binaural room impulse response for localizing the sound as a sound arriving from the direction of the sound image, generating a second output sound signal by convolving the sound information with a second synthesized binaural room impulse response for localizing the sound as a sound arriving from a first direction having a first angle with the direction of the sound image that is greater than 0 degrees and less than 360 degrees, and having a first delay time greater than 0 and a first volume attenuation greater than 0 with respect to the start sound perceived by the first output sound signal, and outputting an output sound signal obtained by synthesizing the generated first output sound signal and the second output sound signal, wherein the first synthesized binaural room impulse response is the binaural room impulse response may be created by synthesizing a direct sound component of a first head impulse response that indicates a first head characteristic of the user and that is associated with a first sound image angle formed by the direction of the sound image and the front direction of the user, and a reverberation sound component of a first binaural room impulse response that indicates a second head characteristic different from the first head characteristic and a spatial characteristic of the space where the user is located and that is associated with the first sound image angle; and the second synthesized binaural room impulse response may be created by synthesizing a direct sound component of a second head impulse response that indicates the first head characteristic and that is associated with a second sound image angle formed by the first direction and the front direction of the user, and a reverberation sound component of a second binaural room impulse response that indicates the second head characteristic and the spatial characteristic and that is associated with the second sound image angle.
[0031] According to this configuration, the second output sound signal arrives from the first direction and is localized as a sound having a first delay time and a first volume attenuation amount, so that the user perceives the reflected sound as the initiation sound reflected by the pseudo-reflection wall. Therefore, the reflected sound having the first delay time and the first volume attenuation amount is perceived together with the initiation sound as a direct sound, improving the out-of-head feeling at the position where the sound image of the direct sound is localized.
[0032] In particular, a first synthesized binaural room impulse response, which is a synthesis of the direct sound component of the first head impulse response and the reverberation sound component of the first binaural room impulse response, is convolved into the sound information, and a second synthesized binaural room impulse response, which is a synthesis of the direct sound component of the second head impulse response and the reverberation sound component of the second binaural room impulse response, is convolved into the sound information. This makes it possible to convolve a binaural room impulse response containing the desired head characteristics and desired spatial characteristics into the sound information, allowing the user to perceive a more three-dimensional sound.
[0033] (10) In the acoustic processing method described in (9) above, the output sound signal may be reproduced using headphones or earphones worn on the user's head.
[0034] According to this configuration, the user can perceive stereoscopic sound more appropriately using headphones or earphones worn on the user's head.
[0035] (11) In the acoustic processing method described in (9) or (10) above, the first angle may be an angle between the direction of the sound image and the direction of the sound image that is greater than 90 degrees and less than 270 degrees.
[0036] According to this configuration, the reflected sound arrives from the opposite direction to the direction from which the initial sound arrives, allowing the user to perceive a more three-dimensional sound.
[0037] (12) In the acoustic processing method according to any one of (9) to (11), the generation of the first output sound signal includes trimming a waveform of the first binaural room impulse response and a waveform of the first head-based impulse response, specifying a replacement time at which a sound pressure difference between the first head-based impulse response and the first binaural room impulse response becomes minimum in a terminal period of the direct sound component of the first head-based impulse response, and replacing the direct sound component of the first binaural room impulse response before the replacement time with the direct sound component of the first head-based impulse response before the replacement time. generating the second output sound signal may include trimming a waveform of the second binaural room impulse response and a waveform of the second head-side impulse response; specifying a replacement time at which a sound pressure difference between the second head-side impulse response and the second binaural room impulse response is minimum in a terminal period of the direct sound component of the second head-side impulse response; and replacing the direct sound component of the second binaural room impulse response before the replacement time with the direct sound component of the second head-side impulse response before the replacement time.
[0038] According to this configuration, the direct sound component of the first binaural room impulse response is replaced with the direct sound component of the first head-based impulse response, thereby easily combining the direct sound component of the first head-based impulse response with the reverberation sound component of the first binaural room impulse response. Also, the direct sound component of the second binaural room impulse response is replaced with the direct sound component of the second head-based impulse response, thereby easily combining the direct sound component of the second head-based impulse response with the reverberation sound component of the second binaural room impulse response.
[0039] (13) In the acoustic processing method described in (12) above, the trimming of the first head-impulse response may include trimming the waveform of the first binaural room impulse response and the waveform of the first head-impulse response so that the time when the amplitude of the first binaural room impulse response is maximum coincides with the time when the amplitude of the first head-impulse response is maximum, and the trimming of the second head-impulse response may include trimming the waveform of the second binaural room impulse response and the waveform of the second head-impulse response so that the time when the amplitude of the second binaural room impulse response is maximum coincides with the time when the amplitude of the second head-impulse response is maximum.
[0040] According to this configuration, the first binaural room impulse response can be appropriately synthesized with the first head-impulse response so as to align the times of the first binaural room impulse response and the first head-impulse response, and the second binaural room impulse response can be appropriately synthesized with the second head-impulse response so as to align the times of the second binaural room impulse response and the second head-impulse response.
[0041] (14) In the acoustic processing method described in any one of (9) to (11) above, generating the first output sound signal may include replacing all amplitudes of direct sound components of the first binaural room impulse response with 0, setting the amplitude of the first binaural room impulse response at time 0 to 1, and convolving the first binaural room impulse response with the first head-on impulse response, and generating the second output sound signal may include replacing all amplitudes of direct sound components of the second binaural room impulse response with 0, setting the amplitude of the second binaural room impulse response at time 0 to 1, and convolving the second binaural room impulse response with the second head-on impulse response.
[0042] According to this configuration, by convolving the first head-based impulse response with the first binaural room impulse response in which the amplitudes of all direct sound components are replaced with 0 and the amplitude at time 0 is set to 1, it is possible to easily combine the direct sound components of the first head-based impulse response and the reverberation sound components of the first binaural room impulse response. Also, by convolving the second head-based impulse response with the second binaural room impulse response in which the amplitudes of all direct sound components are replaced with 0 and the amplitude at time 0 is set to 1, it is possible to easily combine the direct sound components of the second head-based impulse response and the reverberation sound components of the second binaural room impulse response.
[0043] Furthermore, the present disclosure can be realized not only as an acoustic processing method that executes the characteristic processes described above, but also as an acoustic processing device having a characteristic configuration corresponding to the characteristic processes executed by the acoustic processing method. Furthermore, the present disclosure can also be realized as a computer program that causes a computer to execute the characteristic processes included in such an acoustic processing method. Therefore, the following other aspects can also achieve the same effects as the above acoustic processing method.
[0044] (15) Another aspect of the present disclosure is a sound processing device that includes a processor. The processor receives an instruction to start sound processing that causes a user to perceive a reproduced sound as a sound arriving from a predetermined direction in a three-dimensional sound field, localizes a sound image of the starting sound at a starting position outside the user's head, and moves the localized sound image to pass to the side of the user.
[0045] (16) Another aspect of the present disclosure provides an acoustic processing program that receives an instruction to start acoustic processing that causes a user to perceive a reproduced sound as arriving from a predetermined direction in a three-dimensional sound field, and causes a computer to localize a sound image of the starting sound at a starting position outside the user's head and move the localized sound image so that it passes to the side of the user.
[0046] (17) A non-transitory computer-readable recording medium according to another aspect of the present disclosure records the sound processing program described in (16) above.
[0047] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. Note that each of the embodiments described below represents a specific example of the present disclosure. The numerical values, shapes, components, steps, and order of steps shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not described in the independent claims that represent the highest concept are described as optional components. Furthermore, in all embodiments, the respective contents can be combined.
[0048] (Embodiment) FIG. 1 is a schematic diagram showing a first use case of the sound reproducing device according to the present embodiment, and FIG. 2 is a schematic diagram showing a second use case of the sound reproducing device according to the present embodiment.
[0049] The sound reproducing device 100 is a sound presentation device worn on the head of the user 99. Therefore, the sound reproducing device 100 moves integrally with the head of the user 99. For example, the sound reproducing device 100 in this embodiment may be a so-called over-ear headphone type device as shown in FIG. 1, or may be two earplug type devices (in-ear headphone type devices) worn independently on the left and right ears of the user 99 as shown in FIG. 2. These two devices communicate with each other to present sounds for the right ear and the left ear in synchronization.
[0050] The sound reproducing device 100 of the present disclosure is not limited to head-mounted sound reproducing devices such as over-ear headphone devices and inner-ear headphone devices, but may also be applied to sound reproducing devices such as headrest speakers that are not worn by the user 99 and are placed close to both ears of the user 99.
[0051] The sound reproducing device 100 changes the sound presented in accordance with the movement of the head of the user 99, thereby making the user 99 perceive as if he or she were moving his or her head within the three-dimensional sound field. For this reason, as described above, the sound reproducing device 100 moves the three-dimensional sound field in the opposite direction to the movement of the user 99.
[0052] Next, the configuration of the sound reproducing device 100 according to this embodiment will be described with reference to FIGS.
[0053] Fig. 3 is a block diagram showing the functional configuration of the sound reproduction device 100 according to this embodiment. Fig. 4 is a block diagram showing a more detailed functional configuration of the sound processing device 101 according to this embodiment.
[0054] As shown in FIG. 3, the sound reproducing device 100 according to this embodiment includes a sound processing device 101, a communication module 102, a sensor 103, a driver 104, and a start receiving unit 105.
[0055] The start receiving unit 105 receives a user input of an instruction to start acoustic processing that causes the user to perceive a reproduced sound as a sound arriving from a predetermined direction in a three-dimensional sound field. The start receiving unit 105 is configured with a button, a switch, or the like, and switches the sound image from in-head localization to out-of-head localization.
[0056] The sound processing device 101 is an arithmetic device for performing various signal processing in the sound reproduction device 100. The sound processing device 101 includes, for example, a processor and a memory, and performs various functions by the processor executing a program stored in the memory. Note that the sound processing device 101 may be realized, for example, by hardware implementation using a processing circuit, or by execution by the processing circuit of a software program stored in the memory or distributed from an external server, or by a combination of these hardware and software implementations.
[0057] The sound processing device 101 includes an acquisition unit 111, a start sound reproduction unit 112, a sound image control unit 113, a first processing unit 121, a second processing unit 131, and a combiner 150. The acquisition unit 111 will be described later together with the description of the communication module 102, and the combiner 150 will be described later together with the description of the driver 104.
[0058] When a start instruction is received by the start receiving unit 105, the start sound reproducing unit 112 acquires a start sound and reproduces the acquired start sound. The start sound is pre-stored in memory. The start sound is a sound for notifying the user of the start of acoustic processing that causes the user to perceive the reproduced sound as a sound arriving from a predetermined direction in a three-dimensional sound field. For example, the start sound is a sound effect of about 5 seconds.
[0059] The sound image control unit 113 localizes the sound image of the start sound at a start position outside the user's head, and moves the localized sound image so that it passes to the side of the user.
[0060] FIG. 5 is a schematic diagram for explaining the first sound image localization process for moving a sound image localized outside the head in this embodiment.
[0061] Fig. 5 shows a plan view of a virtual three-dimensional sound field from a direction along the up-and-down direction of the head of a user 99. Fig. 5 shows the user 99 with the upward direction of the paper facing forward, and the user 99 is standing upright in a direction perpendicular to the paper. A predetermined direction in which the reproduced sound is localized is set in the direction directly in front of the user 99. Note that start positions 301, 311 and a reproduction position 304 are indicated by black circles.
[0062] The start positions 301 and 311 are located to the sides of the user 99. The start position 301 is located to the left of the user 99.
[0063] The sound image control unit 113 localizes the sound image of the start sound at a start position 301, and moves the sound image from the start position 301 around the user 99 toward a playback position 304 that exists in a predetermined direction to which the playback sound will reach. First, the sound image control unit 113 localizes the sound image inside the user's head at the start position 301 to the left of the user 99 (arrow 302). Then, the sound image control unit 113 moves the sound image clockwise around the user 99 from the start position 301 toward a playback position 304 that exists in the predetermined direction to which the playback sound will reach (arrow 303). This allows the user 99 to hear the start sound moving clockwise from the left toward the front. Then, the user 99 can hear the playback sound from the playback position 304.
[0064] Furthermore, the start position 311 may be located to the right of the user 99. The sound image control unit 113 may localize the sound image of the start sound at the start position 311, and may also move the sound image from the start position 311 around the user 99 toward the playback position 304, which is located in a predetermined direction to which the playback sound will reach. First, the sound image control unit 113 may localize a sound image inside the user's head at the start position 311, which is located to the right of the user 99 (arrow 312). Then, the sound image control unit 113 may move the sound image counterclockwise around the user 99 from the start position 311 toward the playback position 304, which is located in the predetermined direction to which the playback sound will reach (arrow 313). This allows the user 99 to hear the start sound moving counterclockwise from the right toward the front.
[0065] The sides of the user 99 are directions perpendicular to the front direction of the user 99, but the present disclosure is not particularly limited to this. For example, the sides of the user 99 may be within a range of ±45 degrees of the direction perpendicular to the front direction of the user 99.
[0066] FIG. 6 is a schematic diagram for explaining the second sound image localization process for moving a sound image localized outside the head in this embodiment.
[0067] Fig. 6 shows a plan view of a virtual three-dimensional sound field from a direction along the up-and-down direction of the head of a user 99. Fig. 6 shows the user 99 in a posture with the upper side of the paper facing forward, and the user 99 is standing upright in a direction perpendicular to the paper. A predetermined direction in which the reproduced sound is localized is set in the direction directly in front of the user 99. Note that start positions 321, 331 and a reproduction position 324 are indicated by black circles.
[0068] The start positions 321 and 331 are located to the sides of the user 99. The start position 321 is located to the left of the user 99.
[0069] The sound image control unit 113 localizes the sound image of the start sound at a start position 321, and moves the sound image from the start position 321, passing behind the user 99, along the periphery of the user 99 toward a playback position 324 located in a predetermined direction where the playback sound will reach. First, the sound image control unit 113 localizes the sound image inside the user's head at the start position 321 to the left of the user 99 (arrow 322). Then, the sound image control unit 113 moves the sound image counterclockwise from the start position 321, passing behind the user 99, along the periphery of the user 99 toward a playback position 324 located in a predetermined direction where the playback sound will reach (arrow 323). This allows the user 99 to hear the start sound moving counterclockwise from the left to the rear, and also moving counterclockwise from the rear to the front. Then, the user 99 can hear the playback sound from the playback position 304.
[0070] Furthermore, the start position 331 may be located to the right of the user 99. The sound image control unit 113 may localize the sound image of the start sound at the start position 331, and may also move the sound image from the start position 331 along the periphery of the user 99, passing behind the user 99, toward a playback position 324 located in a predetermined direction where the playback sound arrives. First, the sound image control unit 113 may localize a sound image inside the user's head at the start position 331 located to the right of the user 99 (arrow 332). Then, the sound image control unit 113 may move the sound image clockwise from the start position 331 along the periphery of the user 99, passing behind the user 99, toward a playback position 304 located in a predetermined direction where the playback sound arrives (arrow 333). This allows the user 99 to hear the start sound moving clockwise from the right to the rear, and also moving clockwise from the rear to the front.
[0071] FIG. 7 is a schematic diagram for explaining the third sound image localization process for moving a sound image localized outside the head in this embodiment.
[0072] Fig. 7 shows a plan view of a virtual three-dimensional sound field from a direction along the up-and-down direction of the head of a user 99. Fig. 7 shows the user 99 with the upward direction of the paper facing forward, and the user 99 is standing upright in a direction perpendicular to the paper. A predetermined direction in which the reproduced sound is localized is set in the direction directly in front of the user 99. Note that start positions 341, 351 and a reproduction position 344 are indicated by black circles.
[0073] The start positions 341 and 351 are located to the sides of the user 99. The start position 341 is located to the left of the user 99.
[0074] The sound image control unit 113 localizes the sound image of the start sound at a start position 341, moves the sound image from the start position 341 around the user 99 toward a playback position 344 located in a predetermined direction where the playback sound will arrive, and then makes the sound image circle around the user 99 from the playback position 344. First, the sound image control unit 113 localizes the sound image inside the user's head at the start position 341 located to the left of the user 99 (arrow 342). Then, the sound image control unit 113 moves the sound image clockwise around the user 99 from the start position 341 towards a playback position 344 located in a predetermined direction where the playback sound will arrive, and then makes the sound image circle around the user 99 from the playback position 344 in a clockwise direction (arrow 343). This allows the user 99 to hear the start sound moving clockwise from the left toward the front and circling clockwise from the front. The user 99 can then hear the playback sound from the playback position 344.
[0075] Alternatively, the start position 351 may be located to the right of the user 99. The sound image control unit 113 localizes the sound image of the start sound at the start position 351, moves the sound image from the start position 351 around the user 99 toward a playback position 344 located in a predetermined direction where the playback sound will arrive, and then makes the sound image circle around the user 99 from the playback position 344. First, the sound image inside the head is localized at the start position 351 to the right of the user 99 (arrow 352). Then, the sound image control unit 113 moves the sound image counterclockwise from the start position 351 around the user 99 toward the playback position 344 located in a predetermined direction where the playback sound will arrive, and then makes the sound image circle around the user 99 counterclockwise from the playback position 344 (arrow 353). This allows the user 99 to hear the start sound moving counterclockwise from the right toward the front, and circling counterclockwise from the front.
[0076] FIG. 8 is a schematic diagram for explaining the fourth sound image localization process for moving a sound image localized outside the head in this embodiment.
[0077] Fig. 8 shows a plan view of a virtual three-dimensional sound field from a direction along the up-and-down direction of the head of a user 99. Fig. 8 shows the user 99 with the upward direction of the paper facing forward, and the user 99 is standing upright in a direction perpendicular to the paper. A predetermined direction in which the reproduced sound is localized is set in the direction directly in front of the user 99. Note that a start position 361 and a reproduction position 364 are indicated by black circles.
[0078] The start position 361 is a playback position 364 that exists in a predetermined direction in which the playback sound arrives.
[0079] The sound image control unit 113 localizes the sound image of the start sound at a start position 361, and also makes the sound image circle around the user 99 from the start position 361. First, the sound image control unit 113 localizes the sound image inside the head at the start position 361, which is a playback position 364 that exists in a predetermined direction in which the playback sound arrives (arrow 362). Then, the sound image control unit 113 makes the sound image circle around the user 99 counterclockwise from the start position 361 (arrow 363). This allows the user 99 to hear the start sound circling counterclockwise from the front. Then, the user 99 can hear the playback sound from the playback position 364.
[0080] Furthermore, the sound image control unit 113 may make the sound image circle clockwise around the user 99 from the start position 361, which is the playback position 364 (arrow 373). This allows the user 99 to hear the starting sound circling clockwise from the front.
[0081] The difference in sound pressure characteristics and phase characteristics between the two ears becomes greater when the sound image is to the side of the user 99 than when the sound image is in front of the user 99, making it easier to localize the sound image. According to this embodiment, the sound image of the starting sound passes to the side of the user 99, so the effect of out-of-head localization of the sound image can be conveyed to the user 99 in an easily understandable manner.
[0082] The sound image control unit 113 outputs a first sound image angle formed by the direction from the user toward the position of the sound image of the start sound and the front direction of the user to the first processing unit 121 and the second processing unit 131. This first sound image angle changes according to the movement of the position of the sound image of the start sound. In FIG. 5, when the angle of the front direction of the user is 0 degrees and the angle increases clockwise, the sound image control unit 113 gradually increases the first sound image angle formed by the direction of the sound image of the start sound and the front direction of the user from 270 degrees at the start position 301 to 360 degrees (0 degrees) at the playback position 304. Alternatively, the sound image control unit 113 gradually decreases the first sound image angle formed by the direction of the sound image of the start sound and the front direction of the user from 90 degrees at the start position 311 to 0 degrees at the playback position 304.
[0083] 6, when the angle of the front direction of the user is 0 degrees and the angle increases clockwise, the sound image control unit 113 gradually decreases the first sound image angle formed between the direction of the sound image of the start sound and the front direction of the user from 270 degrees at the start position 321 to 0 degrees at the playback position 324. Alternatively, the sound image control unit 113 gradually increases the first sound image angle formed between the direction of the sound image of the start sound and the front direction of the user from 90 degrees at the start position 331 to 360 degrees at the playback position 324.
[0084] 7, when the angle of the front direction of the user is 0 degrees and the angle increases clockwise, the sound image control unit 113 gradually increases the first sound image angle formed by the direction of the sound image of the start sound and the front direction of the user from 270 degrees at start position 341 to 360 degrees (0 degrees) at playback position 344, and also gradually increases it from 0 degrees at playback position 344 to 360 degrees (0 degrees) at playback position 344. Alternatively, the sound image control unit 113 gradually decreases the first sound image angle formed by the direction of the sound image of the start sound and the front direction of the user from 90 degrees at start position 351 to 0 degrees at playback position 344, and also gradually decreases it from 360 degrees at playback position 344 to 0 degrees at playback position 344.
[0085] 8, when the angle of the front direction of the user is 0 degrees and the angle increases clockwise, the sound image control unit 113 gradually decreases the first sound image angle formed by the direction of the sound image of the start sound and the front direction of the user from 360 degrees at the start position 361 to 0 degrees at the playback position 364. Alternatively, the sound image control unit 113 gradually increases the first sound image angle formed by the direction of the sound image of the start sound and the front direction of the user from 0 degrees at the start position 361 to 360 degrees at the playback position 364.
[0086] In this embodiment, the start position is predetermined to the side of the user 99 or to the playback position, but the present disclosure is not limited to this, and the direction in which the effect of out-of-head localization is greatest for the user 99 may be measured in advance, and that direction may be set as the start position.
[0087] The first processing unit 121 generates first output sound signals of the start sound and the playback sound. The first processing unit 121 generates the first output sound signal by convolving sound information including the start sound with a first synthesized binaural room impulse response for localizing the sound included in the information as a sound arriving from the direction of a sound image. The first processing unit 121 also generates the first output sound signal by convolving sound information including the playback sound with a first synthesized binaural room impulse response for localizing the sound included in the information as a sound arriving from a predetermined direction.
[0088] The first synthesized binaural room impulse response is created by synthesizing a direct sound component of a first head-based impulse response that indicates a first head characteristic of the user and is associated with a first sound image angle formed by the direction of the sound image of the start sound and the front direction of the user, and a reverberant sound component of the first binaural room impulse response that indicates a second head characteristic different from the first head characteristic and the spatial characteristic of the space where the user is located and is associated with the first sound image angle. The first head-based impulse response and the first binaural room impulse response are measured using different heads.
[0089] In addition, the first synthesized binaural room impulse response is created by synthesizing a direct sound component of the first head impulse response that indicates a first head characteristic of the user and that corresponds to a first rotation angle formed by a predetermined direction and the front direction of the user, and a reverberation sound component of the first binaural room impulse response that indicates a second head characteristic different from the first head characteristic and the spatial characteristic of the space in which the user is located and that corresponds to the first rotation angle.
[0090] The first processing unit 121 includes a first signal processing unit 123 , a first synthesis BRIR convolution unit 124 , and a first volume attenuation unit 125 .
[0091] The first signal processing unit 123 generates a first synthesized binaural room impulse response by synthesizing the direct sound component of the first head-based impulse response and the reverberation sound component of the first binaural room impulse response. Details of the first signal processing unit 123 will be described later.
[0092] The first synthesis BRIR convolution unit 124 generates a first output sound signal by convolving the first synthesis binaural room impulse response with sound information including the start sound. The first synthesis BRIR convolution unit 124 also generates a first output sound signal by convolving the first synthesis binaural room impulse response with sound information including the playback sound.
[0093] The first volume attenuation unit 125 attenuates the volume of the first output sound signal in accordance with the angle of the head of the user 99. The first volume attenuation unit 125 attenuates the volume of the first output sound signal by a volume attenuation amount α (third volume attenuation amount). Details of the first volume attenuation unit 125 will be described later.
[0094] The first processing unit 121 performs convolution of a first synthesized binaural room impulse response for localizing the sound in the direction of a sound image on the sound information of the input start sound, and outputs a first output sound signal with attenuated volume via a first volume attenuation unit 125. The first output sound signal is input to a first equalizer (EQ) 122. The sound processing device 101 further includes a first EQ 122. The first EQ 122 adjusts low- and high-frequency sounds of the first output sound signal, and outputs the first output sound signal to the combiner 150.
[0095] Furthermore, the first processing unit 121 performs convolution of a first synthesized binaural room impulse response for localizing the sound in a predetermined direction on the sound information of the input playback sound, and outputs a first output sound signal with attenuated volume via a first volume attenuation unit 125. The first output sound signal is input to a first equalizer (EQ) 122.
[0096] The second processing unit 131 generates a second output sound signal of the first reflected sound. The second processing unit 131 generates the second output sound signal by convolving a second synthesized binaural room impulse response for sound information including the start sound, for localizing the sound included in the information as a sound that arrives from a first direction having a first angle with the direction of the sound image of the start sound that is greater than 0 degrees and less than 360 degrees, and has a first delay time greater than 0 and a first volume attenuation greater than 0 with respect to the start sound perceived by the first output sound signal.
[0097] In addition, the second processing unit 131 generates a second output sound signal by convolving a second synthesized binaural room impulse response for sound information including a reproduced sound, in order to localize the sound included in the information as a sound that arrives from a first direction having a first angle with a predetermined direction that is greater than 0 degrees and less than 360 degrees, and has a first delay time greater than 0 and a first volume attenuation greater than 0 with respect to the reproduced sound perceived by the first output sound signal.
[0098] The second synthesized binaural room impulse response is created by synthesizing a direct sound component of a second head-based impulse response that exhibits a first head characteristic and is associated with a second sound image angle formed by a first direction relative to the direction of the sound image of the start sound and a front direction of the user, and a reverberant sound component of a second binaural room impulse response that exhibits a second head characteristic and a spatial characteristic and is associated with the second sound image angle. The second head-based impulse response and the second binaural room impulse response are measured using different heads.
[0099] In addition, the second synthesized binaural room impulse response is created by synthesizing a direct sound component of a second head impulse response that indicates a first head characteristic and that corresponds to a second rotation angle formed by the first direction relative to the predetermined direction and the front direction of the user, and a reverberation sound component of a second binaural room impulse response that indicates a second head characteristic and a spatial characteristic and that corresponds to the second rotation angle.
[0100] The sound processing device 101 further includes a first angle determination unit 130. The first angle determination unit 130 determines a preset first angle. The first angle determination unit 130 also calculates a second sound image angle between a first direction having a first angle determined with respect to the direction of the sound image of the start sound and a front direction of the user obtained by the acquisition unit 111, and outputs the calculated second sound image angle to the second signal processing unit 133. The first angle determination unit 130 also calculates a second rotation angle between the first direction having a first angle determined with respect to a predetermined direction and a front direction of the user obtained by the acquisition unit 111, and outputs the calculated second rotation angle to the second signal processing unit 133.
[0101] As the sound image of the start sound moves, the first sound image angle and the second sound image angle change. Even if the sound image of the start sound moves, the magnitude of the first angle remains unchanged. Therefore, as the sound image of the start sound moves, the position of the sound image of the first reflected sound also moves. For example, if the angle between the front direction of the user and the direction of the sound image of the start sound is 270 degrees and the first angle is 120 degrees, the angle between the front direction of the user and the first direction is 30 degrees. Then, if the angle between the front direction of the user and the direction of the sound image of the start sound changes to 300 degrees, the angle between the front direction of the user and the first direction changes to 60 degrees.
[0102] The second processing unit 131 includes a second signal processing unit 133 , a second synthesis BRIR convolution unit 134 , a first delay unit 135 , and a second volume attenuation unit 136 .
[0103] The second signal processing unit 133 generates a second synthesized binaural room impulse response by synthesizing the direct sound component of the second head-based impulse response and the reverberation sound component of the second binaural room impulse response. Details of the second signal processing unit 133 will be described later.
[0104] The second synthesis BRIR convolution unit 134 generates a second output sound signal by convolving the second synthesis binaural room impulse response with sound information including the start sound. The second synthesis BRIR convolution unit 134 also generates a second output sound signal by convolving the second synthesis binaural room impulse response with sound information including the playback sound.
[0105] The first delay unit 135 delays the second output sound signal according to a first delay time, which will be described later.
[0106] The second volume attenuation unit 136 attenuates the volume of the second output sound signal. The second volume attenuation unit 136 attenuates the volume of the second output sound signal by a volume attenuation amount β (first volume attenuation amount). Details of the second volume attenuation unit 136 will be described later.
[0107] The second processing unit 131 convolves a second synthesized binaural room impulse response for localizing the sound in a first direction with respect to the sound information of the input start sound, and outputs a second output sound signal with attenuated volume via a second volume attenuation unit 136. The second output sound signal is input to a second equalizer (EQ) 132. The sound processing device 101 further includes a second EQ 132. The second EQ 132 adjusts low- and high-frequency sounds of the second output sound signal, and outputs the second output sound signal to the combiner 150.
[0108] Furthermore, the second processing unit 131 performs convolution of a second synthesized binaural room impulse response for localizing the sound in the first direction on the sound information of the input playback sound, and outputs a second output sound signal with attenuated volume via the second volume attenuation unit 136. The second output sound signal is input to a second equalizer (EQ) 132.
[0109] The communication module 102 is an interface device for accepting input of sound information to the sound reproducing device 100. The communication module 102 includes, for example, an antenna and a signal converter, and receives sound information from an external device via wireless communication. More specifically, the communication module 102 receives, using the antenna, a wireless signal indicating sound information converted into a format for wireless communication, and reconverts the wireless signal into sound information using the signal converter. In this way, the sound reproducing device 100 acquires sound information from the external device via wireless communication. The sound information acquired by the communication module 102 is acquired by the acquisition unit 111. In this way, the sound information is input to the sound processing device 101. Note that communication between the sound reproducing device 100 and the external device may be performed via wired communication.
[0110] The sound processing device 101 also includes a dereverberation processing unit 120 shown in FIG. 4. When generating and synthesizing reflected sounds, if the original sound contains reverberation components, i.e., sound components that are input to the sound collector with a delay due to reflection or the like in the sound collection environment, the effect of improving the out-of-head feeling of the sound due to the synthesis of the reflected sounds is reduced. For this reason, the sound processing device 101 uses the dereverberation processing unit 120 to perform dereverberation processing on the sound contained in the information, reducing the reverberation components contained in the information. By performing dereverberation processing on the start sound or the playback sound to be played and original sound information containing reverberation components, sound information containing, as playback sound, sounds contained in the original sound information other than the reduced reverberation components can be generated and input to the first processing unit 121 and the second processing unit 131. The dereverberation processing unit 120 may be inserted before or after the acquisition unit 111.
[0111] The sound information acquired by the sound reproducing device 100 is encoded in a predetermined format, such as MPEG-H 3D Audio (ISO / IEC 23008-3). As an example, the encoded sound information includes information about the sound reproduced by the sound reproducing device 100 and information about the localization position when the sound image of the sound is localized at a predetermined position in a three-dimensional sound field (i.e., perceived as sound arriving from a predetermined direction), i.e., information about the predetermined direction. For example, the sound information includes information about a plurality of sounds including a first reproduced sound and a second reproduced sound, and the sound images are localized so that the sound images when each sound is reproduced are perceived as sound arriving from a different direction in the three-dimensional sound field.
[0112] This three-dimensional sound, together with an image viewed using a display device, can improve the sense of realism of content being viewed. The sound information may include only information about the reproduced sound. In this case, information about a predetermined direction may be acquired separately. As described above, the sound information includes first sound information about a first reproduced sound and second sound information about a second reproduced sound. However, sound images may be localized at different positions in a three-dimensional sound field by acquiring and simultaneously playing multiple pieces of sound information containing these separately. As described above, the form of the input sound information is not particularly limited, and the sound reproduction device 100 (particularly, the sound processing device 101) may be provided with an acquisition unit 111 that can accommodate various forms of sound information.
[0113] The acquisition unit 111 in this embodiment includes, for example, an encoded sound information input unit, a decoding processing unit, and a sensing information input unit.
[0114] The encoded sound information input unit is a processing unit to which coded (in other words, encoded) sound information acquired by the acquisition unit 111 is input. The encoded sound information input unit outputs the input sound information to the decoding processing unit. The decoding processing unit is a processing unit that decodes (in other words, decodes) the sound information output from the encoded sound information input unit to generate information about a specific sound and information about a specific direction contained in the sound information in a format that can be used for subsequent processing. The sensing information input unit will be described below together with the functions of the sensor 103.
[0115] The sensor 103 is a device for detecting the speed of movement of the head of the user 99. The sensor 103 is configured by combining various sensors used for detecting movement, such as a gyro sensor and an acceleration sensor. In this embodiment, the sensor 103 is built into the sound reproducing device 100, but it may also be built into an external device, such as a three-dimensional image reproducing device that operates in response to the movement of the head of the user 99, similar to the sound reproducing device 100. In this case, the sensor 103 does not need to be included in the sound reproducing device 100. Alternatively, the sensor 103 may be an external imaging device or the like that captures an image of the head movement of the user 99 and processes the captured image to detect the head movement of the user 99.
[0116] The sensor 103 is, for example, fixed integrally to the housing of the sound reproducing device 100 and detects the speed of movement of the housing. After the sound reproducing device 100 including the housing is worn by the user 99, it moves integrally with the head of the user 99, and as a result, the sensor 103 can detect the speed of movement of the user 99's head.
[0117] For example, the sensor 103 may detect the amount of rotation about at least one of three mutually orthogonal axes in three-dimensional space as the rotation axis, or may detect the amount of displacement about at least one of the three axes as the displacement direction, as the amount of movement of the head of the user 99. Furthermore, the sensor 103 may detect both the amount of rotation and the amount of displacement as the amount of movement of the head of the user 99.
[0118] Furthermore, in this embodiment, the sound reproducing device 100 may receive in advance a setting input from the user of a predetermined direction in which sound will arrive. For example, the user presses a setting button while facing forward in the direction in which the sound image of the reproduced sound is to be localized. As a result, the current front direction of the user detected by the sensor 103 may be set to the predetermined direction (0 degrees) in which the reproduced sound will arrive. Then, the acquisition unit 111 may calculate a first rotation angle formed between the predetermined direction and the front direction of the user. Furthermore, the sound image control unit 113 may determine a first sound image angle formed between the direction of the sound image of the start sound and the front direction of the user.
[0119] The sensing information input unit of the acquisition unit 111 acquires the movement speed of the user 99's head from the sensor 103. More specifically, the sensing information input unit acquires the amount of head movement of the user 99 detected by the sensor 103 per unit time as the movement speed. In this way, the sensing information input unit acquires at least one of the rotation speed and the displacement speed as the sensing result from the sensor 103. The amount of head movement of the user 99 acquired here is used to determine the coordinates and orientation of the user 99 in a three-dimensional sound field. The sound reproduction device 100 determines the relative position of the sound image based on the determined coordinates and orientation of the user 99, and reproduces the sound. The acquisition unit 111 calculates a first rotation angle formed by a predetermined direction in which the reproduced sound arrives and the front direction of the user 99 obtained by the sensor 103, and outputs the calculated first rotation angle to the first signal processing unit 123. Furthermore, the sound image control unit 113 outputs to the first signal processing unit 123 a first sound image angle formed by the direction in which the starting sound arrives and the front direction of the user 99 obtained by the sensor 103 .
[0120] Furthermore, in this embodiment, the sensing result acquired from the sensor 103 by the sensing information input unit of the acquisition unit 111 is used to control the volume attenuation amounts of the first volume attenuation unit 125 and the second volume attenuation unit 136. In other words, the volume attenuation amounts of the first volume attenuation unit 125 and the second volume attenuation unit 136 change automatically according to the sensing result. This is because, when the user 99 faces the direction of a reflected sound, if the reflected sound is clearly coming from that direction, the user 99 may feel uncomfortable. Therefore, when the user 99 turns his / her head, the second volume attenuation unit 136 controls the volume of the reflected sound to be attenuated as the front direction of the user 99 approaches the direction of the reflected sound. At the same time, the first volume attenuation unit 125 amplifies the volume of the reproduced sound (reduces the volume attenuation amount) so that the overall volume does not change. In other words, the first volume attenuation unit 125 in the first processing unit 121 decreases the volume attenuation amount α when the volume attenuation amount β of the second volume attenuation unit 136 in the second processing unit 131 increases, and increases the volume attenuation amount α when the volume attenuation amount β of the second volume attenuation unit 136 in the second processing unit 131 decreases.
[0121] Fig. 9 is a diagram for explaining volume attenuation according to this embodiment. Fig. 9 shows the volume attenuation amount α (broken line) of the first volume attenuation unit 125 and the volume attenuation amount β (solid line) of the second volume attenuation unit 136 relative to the rotation angle (yaw angle) when the head of the user 99 rotates around an axis parallel to the up-down direction of the head of the user 99. Note that the first angle here is set to 120 degrees. Here, the volume attenuation amount α of the first volume attenuation unit 125 and the volume attenuation amount β of the second volume attenuation unit 136 are calculated based on the following equation (1):
[0122] (α 2 +β 2 ) 1 / 2 =1 (1) In the above formula (1), α denotes the volume attenuation (gain) of the first volume attenuation unit 125, and β denotes the volume attenuation (gain) of the second volume attenuation unit 136. In this example, it can be seen that for a reflected sound set at an angle of 120 degrees with respect to a predetermined direction, the reflected sound disappears when the user 99 turns his / her head to half that angle, 60 degrees. In this way, the sound processing device 101 appropriately changes the volume attenuation of the reproduced sound and the reflected sound so that the reflected sound itself does not cause discomfort. The relationship between the volume attenuation amount α and the volume attenuation amount β described using the above formula (1) is one example, and any relationship may be used as long as the volume attenuation amount β of the reflected sound increases as the user 99 turns his / her head toward the direction of the reflected sound. Furthermore, the above relationship may hold not only for the relationship between the volume attenuation amount α and the volume attenuation amount β, but also for the relationship between the volume attenuation amount α and the volume attenuation amount γ when another reflected sound having a volume attenuation amount γ (described later in a modified example) is generated.
[0123] The combiner 150 outputs an output sound signal obtained by combining the first output sound signal generated by the first processing unit 121 and the second output sound signal generated by the second processing unit 131 to the driver 104. The combiner 150 outputs the output sound signal obtained by adding the first output sound signal and the second output sound signal together. The combiner 150 further generates a waveform signal by performing signal conversion from a digital signal to an analog signal based on the output sound signal, and causes the driver 104 to generate sound waves based on the waveform signal, thereby presenting sound to the user 99. The output output sound signal is reproduced using headphones or earphones worn on the head of the user 99.
[0124] The driver 104 has, for example, a diaphragm and a drive mechanism such as a magnet and a voice coil. The driver 104 operates the drive mechanism in response to a waveform signal, causing the drive mechanism to vibrate the diaphragm. In this way, the driver 104 generates sound waves by vibrating the diaphragm in response to the output sound signal. The sound waves propagate through the air and reach the ears of the user 99, who then perceives the sound.
[0125] When the output sound signal output from the combiner 150 in the above manner is reproduced by the driver 104, a sound field such as that shown in FIG. 10 is formed.
[0126] FIG. 10 is a diagram for explaining the arrival direction of a reproduced sound output by the sound processing device 101 according to this embodiment.
[0127] Fig. 10 shows a plan view of a virtual three-dimensional sound field from a direction along the up-and-down direction of the head of a user 99. Fig. 10 shows the user 99 in a posture with the upper side of the paper facing forward, and the user 99 is standing upright in a direction perpendicular to the paper. A predetermined direction in which the reproduced sound is localized is set in the direction directly in front of the user 99. Note that position P1 where the reproduced sound is localized is shown as a black circle, and a virtual speaker is also shown.
[0128] As shown in FIG. 10, the first reflected sound is localized in a direction that forms a first angle clockwise from the predetermined direction (position P2).
[0129] Furthermore, the dashed-dotted lines extending to the left and right of the user 99 in FIG. 10 indicate a virtual boundary plane dividing the user 99's head into the front and back. This boundary plane may be a plane along the user's 99 ear canal, a plane passing through the rearmost point of the user's 99 auricle, or simply a plane passing through the center of gravity of the user's 99 head. It is known that there is a difference in the ease of hearing sounds before and after such a boundary plane, i.e., in front of and behind the user 99. When localizing reflected sounds, it is effective for the second processing unit 131 to localize the reflected sounds on the rear side of the user 99 to make the presence of the reflected sounds less noticeable. Therefore, it is preferable that the first angle be set to an angle between the predetermined direction and the predetermined direction that is greater than 90 degrees and less than 270 degrees.
[0130] The above-described first angle, first delay time, and first volume attenuation amount have been described as values that are preset by the sound processing device 101 or values that change according to the sensing result of the sensor 103, but at least one of these may be configured to be adjustable by a value arbitrarily input by the user 99. In other words, the sound processing device 101 may accept an input from the user 99 to adjust at least one of the first angle, the first delay time, and the first volume attenuation amount.
[0131] In the above description, the first reflected sound of the reproduced sound is described, but the first reflected sound of the start sound is localized in a direction having a first angle clockwise from the direction of the sound image of the start sound. When the start sound is reproduced, the first angle is preferably set to an angle within an angle range from the direction of the sound image of the start sound that is greater than 90 degrees and less than 270 degrees.
[0132] Next, the first signal processing unit 123 shown in FIG. 4 will be described in detail.
[0133] FIG. 11 is a block diagram showing the configuration of first signal processing unit 123 according to this embodiment.
[0134] The first signal processing unit 123 shown in FIG. 11 includes a first HRIR database 11, a first BRIR database 12, a first HRIR acquisition unit 13, a first BRIR acquisition unit 14, a first creation unit 15, and a first output unit 16.
[0135] The first HRIR database 11 pre-stores head-related impulse responses (HRIRs) that indicate the characteristics of a human head. A head-related transfer function (HRTF) is a frequency domain representation of changes in the physical characteristics of incident sound waves due to the surroundings of the head. A head-related impulse response is a time domain representation of a head-related transfer function. The head-related impulse response is measured in an anechoic space. Therefore, the head-related impulse response includes direct sound components and almost no reverberant sound components.
[0136] The first HRIR database 11 may store head impulse responses for the left and right channels. The first HRIR database 11 may also store multiple head impulse responses corresponding to multiple head characteristics. The first HRIR database 11 may also store multiple head impulse responses corresponding to multiple angles between the direction from which the sound arrives and the front direction of the user. The multiple angles are, for example, angles obtained by dividing 360 degrees into 2-degree, 5-degree, or 10-degree increments. The direction from which the sound arrives is 0 degrees, and the angle indicates the angle formed clockwise between the direction from which the sound arrives and the front direction of the user.
[0137] The first BRIR database 12 stores in advance binaural room impulse responses (BRIRs) that indicate the characteristics of a person's head and the spatial characteristics of the space in which a person is present. The binaural room impulse responses are expressed in the time domain. The binaural room impulse responses are measured in a reverberant space. Therefore, the binaural room impulse responses include direct sound components and reverberant sound components.
[0138] The first BRIR database 12 may store binaural room impulse responses for the left and right channels. The first BRIR database 12 may also store multiple binaural room impulse responses corresponding to multiple spatial characteristics. For example, the first BRIR database 12 may store multiple binaural room impulse responses for each of multiple spatial sizes. The first BRIR database 12 may also store multiple binaural room impulse responses corresponding to multiple angles between a predetermined direction from which sound arrives and the front direction of the user. The multiple angles are, for example, angles obtained by dividing 360 degrees into 2-degree, 5-degree, or 10-degree increments. The direction from which sound arrives is 0 degrees, and the angle indicates the angle between the predetermined direction from which sound arrives and the front direction of the user in a clockwise direction.
[0139] The first HRIR acquisition unit 13 acquires a first head impulse response that indicates a first head characteristic of the user and is associated with a first sound image angle formed between the direction of the sound image of the start sound and the front direction of the user, from the first HRIR database 11. When a start sound is output, the first HRIR acquisition unit 13 acquires the first sound image angle from the sound image control unit 113. The first head impulse response acquired by the first HRIR acquisition unit 13 may be measured by a measuring device attached to the entrance of the ear canal of the user who is the listener, or may be measured by a measuring device attached to the entrance of the ear canal of a person having head characteristics similar to the user's, or by a measuring device attached to a dummy head.
[0140] Furthermore, the first HRIR acquisition unit 13 acquires a first head impulse response that indicates a first head characteristic of the user and is associated with a first rotation angle formed between a predetermined direction and the front direction of the user from the first HRIR database 11. When a playback sound is output, the first HRIR acquisition unit 13 acquires the first rotation angle from the acquisition unit 111.
[0141] The first BRIR acquisition unit 14 acquires a first binaural room impulse response indicating a second head characteristic different from the first head characteristic and the spatial characteristic of the space where the user is located and associated with a first sound image angle from the first BRIR database 12. When a start sound is output, the first BRIR acquisition unit 14 acquires the first sound image angle from the sound image control unit 113. The first binaural room impulse response acquired by the first BRIR acquisition unit 14 may be measured by a measuring device attached to the entrance of the ear canal of a person other than the user or a measuring device attached to a dummy head in a space having spatial characteristics similar to the spatial characteristics of the space where the user, who is a listener, is located.
[0142] Furthermore, the first BRIR acquisition unit 14 acquires a first binaural room impulse response, which indicates a second head characteristic different from the first head characteristic and a spatial characteristic of the space in which the user is located, and which is associated with the first rotation angle, from the first BRIR database 12. When a playback sound is output, the first BRIR acquisition unit 14 acquires the first rotation angle from the acquisition unit 111.
[0143] The first creation unit 15 creates a first synthesized binaural room impulse response by synthesizing the direct sound component of the first head-based impulse response acquired by the first HRIR acquisition unit 13 and the reverberation sound component of the first binaural room impulse response acquired by the first BRIR acquisition unit 14.
[0144] The first output unit 16 outputs the first synthesized binaural room impulse response created by the first creation unit 15.
[0145] Next, the second signal processing unit 133 shown in FIG. 4 will be described in detail.
[0146] FIG. 12 is a block diagram showing the configuration of second signal processing unit 133 according to this embodiment.
[0147] The second signal processing unit 133 shown in FIG. 12 includes a second HRIR database 21, a second BRIR database 22, a second HRIR acquisition unit 23, a second BRIR acquisition unit 24, a second creation unit 25, and a second output unit 26.
[0148] The configuration of the second HRIR database 21 is the same as the configuration of the first HRIR database 11 shown in FIG. 11, and the configuration of the second BRIR database 22 is the same as the configuration of the first BRIR database 12 shown in FIG.
[0149] The second HRIR acquisition unit 23 acquires a second head impulse response, which indicates the first head characteristic of the user and is associated with a second sound image angle formed by a first direction based on the direction of the sound image of the start sound and the front direction of the user, from the second HRIR database 21. The second HRIR acquisition unit 23 acquires the second sound image angle from the first angle determination unit 130. The second head impulse response acquired by the second HRIR acquisition unit 23 may be measured by a measuring device attached to the entrance of the ear canal of the user himself / herself as a listener, or may be measured by a measuring device attached to the entrance of the ear canal of a person having head characteristics similar to the user's or a measuring device attached to a dummy head.
[0150] Furthermore, the second HRIR acquisition unit 23 acquires a second head impulse response that indicates the first head characteristic of the user and is associated with a second rotation angle formed between a first direction based on the predetermined direction and the front direction of the user, from the second HRIR database 21. The second HRIR acquisition unit 23 acquires the second rotation angle from the first angle determination unit 130. The second head impulse response acquired by the second HRIR acquisition unit 23 may be measured by a measuring device attached to the entrance of the ear canal of the user who is the listener, or may be measured by a measuring device attached to the entrance of the ear canal of a person having head characteristics similar to the user's, or by a measuring device attached to a dummy head.
[0151] The second BRIR acquisition unit 24 acquires a second binaural room impulse response indicating second head characteristics different from the first head characteristics and spatial characteristics of the space where the user is located and associated with a second sound image angle from the second BRIR database 22. The second BRIR acquisition unit 24 acquires the second sound image angle from the first angle determination unit 130. The second binaural room impulse response acquired by the second BRIR acquisition unit 24 may be measured by a measuring device attached to the entrance of the ear canal of a person other than the user or a measuring device attached to a dummy head in a space having spatial characteristics similar to the spatial characteristics of the space where the user, who is a listener, is located.
[0152] Furthermore, the second BRIR acquisition unit 24 acquires a second binaural room impulse response indicating second head characteristics different from the first head characteristics and spatial characteristics of the space where the user is located and associated with the second rotation angle from the second BRIR database 22. The second BRIR acquisition unit 24 acquires the second rotation angle from the first angle determination unit 130. The second binaural room impulse response acquired by the second BRIR acquisition unit 24 may be measured by a measuring device attached to the entrance of the ear canal of a person other than the user or a measuring device attached to a dummy head in a space having spatial characteristics similar to the spatial characteristics of the space where the user, who is a listener, is located.
[0153] The second creation unit 25 creates a second synthesized binaural room impulse response by synthesizing the direct sound component of the second head impulse response acquired by the second HRIR acquisition unit 23 and the reverberation sound component of the second binaural room impulse response acquired by the second BRIR acquisition unit 24.
[0154] The second output unit 26 outputs the second synthesized binaural room impulse response created by the second creation unit 25.
[0155] Here, the first synthesized binaural room impulse response and the second synthesized binaural room impulse response created in this embodiment will be described. Note that the method for creating the first synthesized binaural room impulse response is the same as the method for creating the second synthesized binaural room impulse response, so in the following explanation, only the method for creating the first synthesized binaural room impulse response will be described.
[0156] Fig. 13 is a diagram for explaining a method for creating a first synthesized binaural room impulse response in this embodiment. Fig. 13 shows the first synthesized binaural room impulse responses for the left channel (L) and the right channel (R).
[0157] 13, the first creation unit 15 combines direct sound components 201L and 201R of the first head impulse response and reverberation sound components 202L and 202R of the first binaural room impulse response, thereby creating first combined binaural room impulse responses 203L and 203R.
[0158] FIG. 14 is a block diagram showing the configuration of the first creating unit 15 according to this embodiment.
[0159] The first creating unit 15 shown in FIG. 14 includes a first peak position correcting unit 151, a first identifying unit 152, a first direct sound component substituting unit 153, and a first sound pressure level changing unit 154.
[0160] The first peak position corrector 151 trims the waveform of the first binaural room impulse response and the waveform of the first head-accepting impulse response so that the time when the amplitude of the first binaural room impulse response is maximized coincides with the time when the amplitude of the first head-accepting impulse response is maximized.
[0161] The first determination unit 152 determines a replacement time at which the sound pressure difference between the first head impulse response and the first binaural room impulse response is minimum during the terminal period of the direct sound component of the first head impulse response. That is, the first determination unit 152 determines a replacement time at which the difference between the absolute value of the amplitude of the first head impulse response and the absolute value of the amplitude of the first binaural room impulse response is minimum during the terminal period of the direct sound component of the first head impulse response. The terminal period is, for example, a period from 1.5 msec to 2.0 msec. The terminal period may be, for example, a period from 2.0 msec to 2.5 msec. Alternatively, the terminal period may be any period between 1.5 msec and 2.5 msec. Alternatively, the propagation time of the direct sound may be calculated based on the positions of the sound source and the microphone in the measurement environment, and the terminal period may be determined based on the calculated propagation time.
[0162] The first direct sound component substitution unit 153 substitutes the direct sound component of the first binaural room impulse response before the substitution time with the direct sound component of the first head-based impulse response before the substitution time, thereby combining the direct sound component of the first head-based impulse response and the reverberation sound component of the first binaural room impulse response.
[0163] The first sound pressure level modification unit 154 modifies the sound pressure level of the direct sound component of the first head impulse response before the replacement time according to the sound pressure level of the direct sound component of the first binaural room impulse response before the replacement time.
[0164] Specifically, the first sound pressure level modification unit 154 calculates the average value of the A-weighted sound pressure levels of the direct sound components of the first head impulse response before the replacement time, and also calculates the average value of the A-weighted sound pressure levels of the direct sound components of the first binaural room impulse response before the replacement time. The first sound pressure level modification unit 154 calculates the ratio between the average value of the A-weighted sound pressure levels of the direct sound components of the first binaural room impulse response and the average value of the A-weighted sound pressure levels of the direct sound components of the first head impulse response. The ratio is calculated by dividing the average value of the A-weighted sound pressure levels of the direct sound components of the first binaural room impulse response by the average value of the A-weighted sound pressure levels of the direct sound components of the first head impulse response. The first sound pressure level modification unit 154 multiplies the sound pressure level of the direct sound component of the first head impulse response by the calculated ratio. As a result, the sound pressure level of the direct sound component of the first head-based impulse response before the replacement time is adjusted to the sound pressure level of the direct sound component of the first binaural room impulse response before the replacement time.
[0165] The first sound pressure level change unit 154 may change the sound pressure level of the reverberation sound component of the first binaural room impulse response after the replacement time in accordance with the sound pressure level of the direct sound component of the first head-based impulse response before the replacement time.
[0166] Specifically, the first sound pressure level modification unit 154 calculates the average value of the A-weighted sound pressure levels of the direct sound components of the first head impulse response before the replacement time, and also calculates the average value of the A-weighted sound pressure levels of the direct sound components of the first binaural room impulse response before the replacement time. The first sound pressure level modification unit 154 calculates the ratio between the average value of the A-weighted sound pressure levels of the direct sound components of the first head impulse response and the average value of the A-weighted sound pressure levels of the direct sound components of the first binaural room impulse response. The ratio is calculated by dividing the average value of the A-weighted sound pressure levels of the direct sound components of the first head impulse response by the average value of the A-weighted sound pressure levels of the direct sound components of the first binaural room impulse response. The first sound pressure level modification unit 154 multiplies the calculated ratio by the sound pressure level of the reverberation sound component of the first binaural room impulse response after the replacement time. As a result, the sound pressure level of the reverberation sound component of the first binaural room impulse response after the replacement time is made equal to the sound pressure level of the direct sound component of the first head-based impulse response before the replacement time.
[0167] FIG. 15 is a block diagram showing the configuration of the second creation unit 25 according to this embodiment.
[0168] The second creation unit 25 shown in FIG. 15 includes a second peak position correction unit 251, a second identification unit 252, a second direct sound component substitution unit 253, and a second sound pressure level change unit 254.
[0169] The second peak position corrector 251 trims the waveform of the second binaural room impulse response and the waveform of the second head-accepting impulse response so that the time when the amplitude of the second binaural room impulse response is maximized coincides with the time when the amplitude of the second head-accepting impulse response is maximized.
[0170] The second determination unit 252 determines a replacement time at which the sound pressure difference between the second head impulse response and the second binaural room impulse response is minimum during the terminal period of the direct sound component of the second head impulse response. That is, the second determination unit 252 determines a replacement time at which the difference between the absolute value of the amplitude of the second head impulse response and the absolute value of the amplitude of the second binaural room impulse response is minimum during the terminal period of the direct sound component of the second head impulse response. The terminal period is, for example, a period from 1.5 msec to 2.0 msec. The terminal period may be, for example, a period from 2.0 msec to 2.5 msec. Alternatively, the terminal period may be any period between 1.5 msec and 2.5 msec. Alternatively, the propagation time of the direct sound may be calculated based on the positions of the sound source and the microphone in the measurement environment, and the terminal period may be determined based on the calculated propagation time.
[0171] The second direct sound component substitution unit 253 substitutes the direct sound component of the second binaural room impulse response before the substitution time with the direct sound component of the second head-based impulse response before the substitution time, thereby combining the direct sound component of the second head-based impulse response and the reverberation component of the second binaural room impulse response.
[0172] The second sound pressure level change unit 254 changes the sound pressure level of the direct sound component of the second head impulse response before the replacement time according to the sound pressure level of the direct sound component of the second binaural room impulse response before the replacement time.
[0173] Specifically, the second sound pressure level modification unit 254 calculates the average value of the A-weighted sound pressure levels of the direct sound components of the second head impulse response before the replacement time, and also calculates the average value of the A-weighted sound pressure levels of the direct sound components of the second binaural room impulse response before the replacement time. The second sound pressure level modification unit 254 calculates the ratio between the average value of the A-weighted sound pressure levels of the direct sound components of the second binaural room impulse response and the average value of the A-weighted sound pressure levels of the direct sound components of the second head impulse response. The ratio is calculated by dividing the average value of the A-weighted sound pressure levels of the direct sound components of the second binaural room impulse response by the average value of the A-weighted sound pressure levels of the direct sound components of the second head impulse response. The second sound pressure level modification unit 254 multiplies the sound pressure level of the direct sound component of the second head impulse response by the calculated ratio. As a result, the sound pressure level of the direct sound component of the second head-based impulse response before the replacement time is adjusted to the sound pressure level of the direct sound component of the second binaural room impulse response before the replacement time.
[0174] The second sound pressure level change unit 254 may change the sound pressure level of the reverberation sound component of the second binaural room impulse response after the replacement time in accordance with the sound pressure level of the direct sound component of the second head-based impulse response before the replacement time.
[0175] Specifically, the second sound pressure level modification unit 254 calculates the average value of the A-weighted sound pressure levels of the direct sound components of the second head impulse response before the replacement time, and also calculates the average value of the A-weighted sound pressure levels of the direct sound components of the second binaural room impulse response before the replacement time. The second sound pressure level modification unit 254 calculates the ratio between the average value of the A-weighted sound pressure levels of the direct sound components of the second head impulse response and the average value of the A-weighted sound pressure levels of the direct sound components of the second binaural room impulse response. The ratio is calculated by dividing the average value of the A-weighted sound pressure levels of the direct sound components of the second head impulse response by the average value of the A-weighted sound pressure levels of the direct sound components of the second binaural room impulse response. The second sound pressure level modification unit 254 multiplies the sound pressure level of the reverberation sound component of the second binaural room impulse response after the replacement time by the calculated ratio. As a result, the sound pressure level of the reverberation sound component of the second binaural room impulse response after the replacement time is made equal to the sound pressure level of the direct sound component of the second head-based impulse response before the replacement time.
[0176] In the present embodiment, when a start instruction is received by the start receiving unit 105, the start sound reproducing unit 112 reproduces the start sound, but the present disclosure is not particularly limited to this. When a start instruction is received by the start receiving unit 105, the sound image control unit 113 may localize the sound image of the playback sound being reproduced when the start instruction is received at the start position as the sound image of the start sound.
[0177] Next, the operation of the above-described sound reproducing device 100 will be described with reference to FIG.
[0178] FIG. 16 is a flowchart showing the operation of the sound processing device 101 according to this embodiment.
[0179] First, in step S1, the start receiving unit 105 receives an input from a user of an instruction to start acoustic processing that causes the user to perceive a reproduced sound as a sound arriving from a predetermined direction in a three-dimensional sound field.
[0180] Next, in step S2, the start sound reproducing unit 112 reproduces the start sound. When the start instruction is accepted by the start accepting unit 105, the start sound reproducing unit 112 acquires sound information including the start sound from the memory.
[0181] Next, in step S3, the sound image control unit 113 determines the position of the sound image of the start sound. Specifically, the sound image control unit 113 determines a start position for initially localizing the sound image of the start sound outside the head, and outputs a first sound image angle formed by the direction of the determined start position and the front direction of the user to the first signal processing unit 123 and the first angle determination unit 130.
[0182] Next, in step S4, the first processing unit 121 generates a first output sound signal by convolving the sound information including the start sound with a first synthesized binaural room impulse response for localizing the sound included in the information as a sound arriving from the direction of the sound image of the start sound.
[0183] Next, in step S5, the second processing unit 131 generates a second output sound signal by convolving the sound information including the start sound with a second synthesized binaural room impulse response for localizing the sound included in the information as a sound that arrives from a first direction and has a first delay time greater than 0 and a volume attenuation β (first volume attenuation) greater than 0 relative to the reproduced sound perceived by the first output sound signal.
[0184] The processing of steps S4 and S5 may be performed in a reversed order or in parallel.
[0185] Next, in step S6, the combiner 150 combines the first output sound signal generated by the first processing unit 121 and the second output sound signal generated by the second processing unit 131, and outputs the combined output sound signal.
[0186] When the output sound signal output in this manner is reproduced by the driver 104, the reflected sound is superimposed on the starting sound, and is perceived by the user 99 as a three-dimensional sound. In particular, since only one reflected sound is generated, no large-scale computing device is required, and the user 99 can perceive effective stereophonic sound.
[0187] Next, in step S7, the sound image control unit 113 determines whether or not the position of the sound image of the starting sound is a reproduction position that exists in a predetermined direction to which the reproduced sound arrives.
[0188] Here, if it is determined that the position of the sound image of the start sound is not the playback position (NO in step S7), the process returns to step S3, and the sound image control unit 113 determines the position of the sound image of the start sound by moving it from the previously determined position of the sound image of the start sound. Specifically, the sound image control unit 113 determines the position of the sound image by moving it from the start position toward the playback position along the periphery of the user. For example, in the example of Fig. 5, the first sound image angle between the direction of the start position and the front direction of the user is 270 degrees, and the sound image control unit 113 increases the angle from the start position to the playback position by a predetermined angle (for example, 2 degrees).
[0189] On the other hand, if it is determined that the position of the sound image of the start sound is the playback position (YES in step S7), the process ends.
[0190] After the output of the start sound has finished, the output of the reproduced sound begins. First, when the operation of the sound reproducing device 100 starts, the acquisition unit 111 acquires original sound information via the communication module 102. The original sound information includes reverberation components in addition to the reproduced sound. Therefore, the dereverberation processing unit 120 generates sound information including the reproduced sound with the reverberation components reduced.
[0191] When the reproduced sound is output, the processes of steps S4 to S6 in FIG. 16 are carried out.
[0192] First, the first processing unit 121 generates a first output sound signal by convolving the sound information with a first synthesized binaural room impulse response for localizing the sound included in the information as a sound arriving from a predetermined direction.
[0193] Next, the second processing unit 131 generates a second output sound signal by convolving the sound information including the reproduced sound with a second synthesized binaural room impulse response for localizing the sound included in the information as a sound that arrives from a first direction and has a first delay time greater than 0 and a volume attenuation β (first volume attenuation) greater than 0 relative to the reproduced sound perceived by the first output sound signal.
[0194] Next, the combiner 150 combines the first output sound signal generated by the first processing unit 121 and the second output sound signal generated by the second processing unit 131, and outputs the combined output sound signal.
[0195] The output sound signal thus output is reproduced by the driver 104, and the reproduced sound is superimposed with the reflected sound, which is perceived as a three-dimensional sound by the user 99. In particular, since only one reflected sound is generated, no large-scale computing device is required, and the user 99 can perceive effective stereophonic sound.
[0196] Next, the first output sound signal generation process by the first processing unit 121 in step S4 of FIG. 16 will be described.
[0197] FIG. 17 is a flowchart for explaining the first output sound signal generation process by the first processing unit 121 according to this embodiment.
[0198] First, in step S11, the first HRIR acquisition unit 13 acquires a first head impulse response that indicates a first head characteristic of the user and is associated with a first sound image angle formed by the direction of the sound image and the front direction of the user from the first HRIR database 11. Note that the first HRIR acquisition unit 13 may acquire a first head impulse response for the left channel and a first head impulse response for the right channel.
[0199] Next, in step S12, the first BRIR acquisition unit 14 acquires a first binaural room impulse response indicating a second head characteristic different from the first head characteristic and a spatial characteristic of the space where the user is located and associated with the first sound image angle from the first BRIR database 12. Note that the first BRIR acquisition unit 14 may acquire a first binaural room impulse response for the left channel and a first binaural room impulse response for the right channel.
[0200] Next, in step S13, the first creation unit 15 executes a first synthesized binaural room impulse response creation process to create a first synthesized binaural room impulse response by combining the direct sound component of the first head-related impulse response acquired by the first HRIR acquisition unit 13 and the reverberation sound component of the first binaural room impulse response acquired by the first BRIR acquisition unit 14. The first synthesized binaural room impulse response creation process will be described later with reference to Fig. 19. The first creation unit 15 may also create a first synthesized binaural room impulse response for the left channel and a first synthesized binaural room impulse response for the right channel.
[0201] Next, in step S14, the first output unit 16 outputs the first synthesized binaural room impulse response. Note that the first output unit 16 may output the first synthesized binaural room impulse response for the left channel and the first synthesized binaural room impulse response for the right channel.
[0202] Next, in step S15, the first synthesized BRIR convolution unit 124 convolves the first synthesized binaural room impulse response with the sound information including the start sound, thereby generating a first output sound signal.
[0203] Next, in step S16, the first volume attenuation unit 125 attenuates the volume of the first output sound signal by the volume attenuation amount α (third volume attenuation amount). Then, the first output sound signal is output to the combiner 150 via the first equalizer (EQ) 122.
[0204] Next, the second output sound signal generation process by the second processing unit 131 in step S5 of FIG. 16 will be described.
[0205] FIG. 18 is a flowchart for explaining the second output sound signal generation process by the second processing unit 131 according to this embodiment.
[0206] First, in step S21, the second HRIR acquisition unit 23 acquires a second head impulse response that indicates the first head characteristic of the user and is associated with a second sound image angle formed by the first direction and the front direction of the user from the second HRIR database 21. The second HRIR acquisition unit 23 may acquire a second head impulse response for the left channel and a second head impulse response for the right channel.
[0207] Next, in step S22, the second BRIR acquisition unit 24 acquires a second binaural room impulse response indicating a second head characteristic different from the first head characteristic and a spatial characteristic of the space where the user is located and associated with a second sound image angle from the second BRIR database 22. Note that the second BRIR acquisition unit 24 may acquire a second binaural room impulse response for the left channel and a second binaural room impulse response for the right channel.
[0208] Next, in step S23, the second creation unit 25 executes a second synthesized binaural room impulse response creation process to create a second synthesized binaural room impulse response by combining the direct sound component of the second head-related impulse response acquired by the second HRIR acquisition unit 23 and the reverberation sound component of the second binaural room impulse response acquired by the second BRIR acquisition unit 24. The second synthesized binaural room impulse response creation process will be described later with reference to Fig. 19. The second creation unit 25 may also create a second synthesized binaural room impulse response for the left channel and a second synthesized binaural room impulse response for the right channel.
[0209] Next, in step S24, the second output unit 26 outputs the second synthesized binaural room impulse response. Note that the second output unit 26 may output the second synthesized binaural room impulse response for the left channel and the second synthesized binaural room impulse response for the right channel.
[0210] Next, in step S25, the second synthesized BRIR convolution unit 134 convolves the second synthesized binaural room impulse response with the sound information including the start sound, thereby generating a second output sound signal.
[0211] Next, in step S26, the first delay unit 135 delays the second output sound signal according to the first delay time.
[0212] Next, in step S27, the second volume attenuation unit 136 attenuates the volume of the second output sound signal by the volume attenuation amount β (first volume attenuation amount). Then, the second output sound signal is output to the combiner 150 via the second equalizer (EQ) 132.
[0213] In this way, the second output sound signal arrives from the first direction and is localized as a sound having the first delay time and the first volume attenuation amount, and is therefore perceived by the user as a reflected sound resulting from the initiation sound being reflected by a pseudo-reflection wall. Therefore, the reflected sound having the first delay time and the first volume attenuation amount is perceived together with the initiation sound as a direct sound, thereby improving the out-of-head feeling at the position where the sound image of the direct sound is localized.
[0214] In particular, a first synthesized binaural room impulse response, which is a synthesis of the direct sound component of the first head impulse response and the reverberation sound component of the first binaural room impulse response, is convolved into the sound information, and a second synthesized binaural room impulse response, which is a synthesis of the direct sound component of the second head impulse response and the reverberation sound component of the second binaural room impulse response, is convolved into the sound information. This makes it possible to convolve a binaural room impulse response containing the desired head characteristics and desired spatial characteristics into the sound information, allowing the user to perceive a more three-dimensional sound.
[0215] Next, a description will be given of the first synthesized binaural room impulse response creation process in step S13 of Fig. 17 and the second synthesized binaural room impulse response creation process in step S23 of Fig. 18. Note that the second synthesized binaural room impulse response creation process is the same as the first synthesized binaural room impulse response creation process, so only the first synthesized binaural room impulse response creation process will be described below.
[0216] FIG. 19 is a flowchart illustrating the first synthesized binaural room impulse response creation process by the first creation unit 15 according to the embodiment of the present disclosure.
[0217] First, in step S31, the first peak position correction unit 151 trims the waveform of the first binaural room impulse response and the waveform of the first head-induced impulse response so that the time when the amplitude of the first binaural room impulse response is maximum coincides with the time when the amplitude of the first head-induced impulse response is maximum.
[0218] Next, in step S32, the first identifying unit 152 identifies a replacement time at which the sound pressure difference between the first head-based impulse response and the first binaural room impulse response is minimum in the terminal period of the direct sound component of the first head-based impulse response.
[0219] Next, in step S33, the first direct sound component substitution unit 153 substitutes the direct sound component of the first binaural room impulse response before the substitution time with the direct sound component of the first head impulse response before the substitution time. In this way, the first direct sound component substitution unit 153 creates a first synthesized binaural room impulse response that combines the direct sound component of the first head impulse response before the substitution time and the reverberation sound component of the first binaural room impulse response after the substitution time.
[0220] Next, in step S34, the first sound pressure level change unit 154 changes the sound pressure level of the direct sound component of the first head impulse response before the replacement time according to the sound pressure level of the direct sound component of the first binaural room impulse response before the replacement time.
[0221] In addition, when a starting sound is played and a first synthesized binaural room impulse response is created for each of a plurality of head angles, the first sound pressure level modification unit 154 may use the ratio calculated when creating the first synthesized binaural room impulse response corresponding to the direction of the starting position where the sound arrives to modify the sound pressure level of the direct sound component of the first head impulse response or the sound pressure level of the reverberant sound component of the first binaural room impulse response corresponding to other angles.
[0222] Furthermore, when a playback sound is played and a first synthesized binaural room impulse response is created for each of a plurality of head angles, the first sound pressure level modification unit 154 may use the ratio calculated when creating the first synthesized binaural room impulse response corresponding to a predetermined direction (0 degrees) from which the sound arrives to modify the sound pressure level of the direct sound component of the first head impulse response or the sound pressure level of the reverberant sound component of the first binaural room impulse response corresponding to other angles.
[0223] The above first synthesized binaural room impulse response creation process creates a first synthesized binaural room impulse response by combining the direct sound component of the first head impulse response and the reverberation sound component of the first binaural room impulse response. Also, the second synthesized binaural room impulse response creation process creates a second synthesized binaural room impulse response by combining the direct sound component of the second head impulse response and the reverberation sound component of the second binaural room impulse response.
[0224] In the above-described first synthesized binaural room impulse response creation process, a first synthesized binaural room impulse response may be created for each of the left channel and the right channel. Also, a first synthesized binaural room impulse response may be created for each of a plurality of head angles. Furthermore, when the angle of the front direction of the user's head relative to the sound arrival direction is acquired, a first synthesized binaural room impulse response corresponding to the acquired angle may be created.
[0225] In this way, by replacing the direct sound component of the first binaural room impulse response with the direct sound component of the first head-based impulse response, the direct sound component of the first head-based impulse response and the reverberation sound component of the first binaural room impulse response can be synthesized. Also, by replacing the direct sound component of the second binaural room impulse response with the direct sound component of the second head-based impulse response, the direct sound component of the second head-based impulse response and the reverberation sound component of the second binaural room impulse response can be synthesized.
[0226] Furthermore, the direct sound components of the first and second binaural room impulse responses include head characteristics, while the reverberation sound components of the first and second binaural room impulse responses include spatial characteristics. While the reverberation sound components of the first and second binaural room impulse responses also include head characteristics, their influence is quite small. Therefore, by replacing the direct sound components of the first and second binaural room impulse responses that include desired spatial characteristics with the direct sound components of the first and second head-based impulse responses that include desired head characteristics, it is possible to artificially reproduce the first and second binaural room impulse responses that include desired head characteristics and desired spatial characteristics.
[0227] FIG. 20 is a diagram showing an example of the first binaural room impulse response and the first head-based impulse response in the present embodiment.
[0228] Note that, in the following, only the first binaural room impulse response, the first head-based impulse response, and the first synthesized binaural room impulse response will be described, but the second binaural room impulse response, the second head-based impulse response, and the second synthesized binaural room impulse response are similar to the first binaural room impulse response, the first head-based impulse response, and the first synthesized binaural room impulse response.
[0229] In Fig. 20, the vertical axis represents amplitude, the horizontal axis represents time (msec), the solid lines represent first binaural room impulse responses 211L and 211R, and the dashed lines represent first head impulse responses 212L and 212R. The waveform diagram on the left side of Fig. 20 represents the first binaural room impulse response 211L and the first head impulse response 212L of the left channel when the predetermined direction (0 degrees) from which sound arrives coincides with the front direction of the user, and the waveform diagram on the right side of Fig. 20 represents the first binaural room impulse response 211R and the first head impulse response 212R of the right channel when the predetermined direction (0 degrees) from which sound arrives coincides with the front direction of the user.
[0230] As shown in FIG. 20, the first peak position correction unit 151 trims the waveforms of the first binaural room impulse responses 211L and 211R and the waveforms of the first head impulse responses 212L and 212R so that the time when the amplitudes of the first binaural room impulse responses 211L and 211R are maximum coincides with the time when the amplitudes of the first head impulse responses 212L and 212R are maximum.
[0231] Specifically, the first peak position correction unit 151 calculates the average times t_b and t_h of the left and right maximum amplitude positions at an angle of 0 degrees for each of the first binaural room impulse response and the first head impulse response. Then, the first peak position correction unit 151 performs signal processing on either the first binaural room impulse response or the first head impulse response so that the average time t_b of the first binaural room impulse response coincides with the average time t_h of the first head impulse response.
[0232] That is, if t_b = t_h, the first peak position correction unit 151 does not perform signal processing.
[0233] Also, if t_b < t_h, the first peak position correction unit 151 adds 0 to the beginning of the first binaural room impulse response so that t_b = t_h. Alternatively, the first peak position correction unit 151 cuts the beginning of the first head impulse response so that t_b = t_h.
[0234] Also, if t_b > t_h, the first peak position correction unit 151 adds 0 to the beginning of the first head impulse response so that t_b = t_h. Alternatively, the first peak position correction unit 151 cuts the beginning of the first binaural room impulse response so that t_b = t_h.
[0235] Furthermore, the first identification unit 152 identifies replacement times 213L, 213R at which the sound pressure difference between the first head impulse responses 212L, 212R and the first binaural room impulse responses 211L, 211R is minimum in the terminal period (for example, 1.5 msec to 2.0 msec) of the direct sound components of the first head impulse responses 212L, 212R.
[0236] The parts before the replacement times 213L and 213R indicate direct sound components, and the parts after the replacement times 213L and 213R indicate reverberation sound components. In Fig. 20, the parts enclosed in square frames indicate the direct sound components of the first binaural room impulse responses 211L and 211R and the first head impulse responses 212L and 212R. The first direct sound component replacement unit 153 replaces the direct sound components of the first binaural room impulse responses 211L and 211R before the replacement times 213L and 213R with the direct sound components of the first head impulse responses 212L and 212R before the replacement times 213L and 213R.
[0237] FIG. 21 is a diagram showing an example of the first synthesized binaural room impulse response in the present embodiment.
[0238] In Fig. 21, the vertical axis represents amplitude, the horizontal axis represents time (msec), the solid lines represent first synthesized binaural room impulse responses 214L and 214R, and the dashed lines represent first binaural room impulse responses 211L and 211R. The waveform diagram on the left side of Fig. 21 represents first synthesized binaural room impulse response 214L of the left channel when the predetermined direction (0 degrees) from which sound arrives coincides with the front direction of the user, and the waveform diagram on the right side of Fig. 21 represents first synthesized binaural room impulse response 214R of the right channel when the predetermined direction (0 degrees) from which sound arrives coincides with the front direction of the user.
[0239] The first synthesized binaural room impulse responses 214L and 214R are created by replacing the direct sound components of the first binaural room impulse responses 211L and 211R with the direct sound components of the first head-based impulse responses 212L and 212R. Therefore, the first synthesized binaural room impulse responses 214L and 214R are composed of the direct sound components of the first head-based impulse responses 212L and 212R and the reverberation sound components of the first binaural room impulse responses 211L and 211R.
[0240] 21, the first sound pressure level modification unit 154 modifies the sound pressure levels of the direct sound components of the first synthesized binaural room impulse responses 214L, 214R corresponding to the direct sound components of the first head impulse responses 212L, 212R in accordance with the sound pressure levels of the direct sound components of the first binaural room impulse responses 211L, 211R. As a result, the amplitudes of the direct sound components of the first synthesized binaural room impulse responses 214L, 214R shown in Fig. 21 are smaller than the amplitudes of the direct sound components of the first head impulse responses 212L, 212R shown in Fig. 16.
[0241] FIG. 22 is a diagram illustrating the process of aligning the maximum amplitude position of the first binaural room impulse response and the maximum amplitude position of the first head-based impulse response in the present embodiment.
[0242] In Fig. 22, the vertical axis represents amplitude, the horizontal axis represents time (msec), the solid line represents the first binaural room impulse response, and the dashed line represents the first head-impulse response. The first and third waveform diagrams from the top of Fig. 22 show the first binaural room impulse response and the first head-impulse response of the left channel when the predetermined direction (0 degrees) from which sound arrives coincides with the front direction of the user, and the second and fourth waveform diagrams from the top of Fig. 22 show the first binaural room impulse response and the first head-impulse response of the right channel when the predetermined direction (0 degrees) from which sound arrives coincides with the front direction of the user.
[0243] As shown in FIG. 22, the maximum amplitude position of the first binaural room impulse response in the left channel may differ from the maximum amplitude position of the first binaural room impulse response in the right channel, and the maximum amplitude position of the first head-based impulse response in the left channel may differ from the maximum amplitude position of the first head-based impulse response in the right channel.
[0244] Therefore, the first peak position correction unit 151 may calculate the average time of the time when the amplitude of the first binaural room impulse response of the left channel is maximum and the time when the amplitude of the first binaural room impulse response of the right channel is maximum, and may also calculate the average time of the time when the amplitude of the first head-type impulse response of the left channel is maximum and the time when the amplitude of the first head-type impulse response of the right channel is maximum.The first peak position correction unit 151 may then trim the waveforms of the first binaural room impulse response and the first head-type impulse response so that the average time when the amplitude of the first binaural room impulse response is maximum coincides with the average time when the amplitude of the first head-type impulse response is maximum.
[0245] Furthermore, the first peak position correction unit 151 may calculate an average maximum amplitude of the maximum amplitude of the first binaural room impulse response of the left channel and the maximum amplitude of the first binaural room impulse response of the right channel, and may also calculate an average maximum amplitude of the maximum amplitude of the first head-impulse response of the left channel and the maximum amplitude of the first head-impulse response of the right channel. Then, the first peak position correction unit 151 may calculate a ratio between the average maximum amplitude of the first head-impulse response and the average maximum amplitude of the first binaural room impulse response. The ratio is calculated by dividing the average maximum amplitude of the first head-impulse response by the average maximum amplitude of the first binaural room impulse response. Then, the first peak position correction unit 151 may multiply the first binaural room impulse response by the calculated ratio. This aligns the amplitude of the first binaural room impulse response with the amplitude of the first head-impulse response.
[0246] The first peak position corrector 151 may calculate a ratio between the average maximum amplitude of the first binaural room impulse response and the average maximum amplitude of the first head-impulse response. The ratio is calculated by dividing the average maximum amplitude of the first binaural room impulse response by the average maximum amplitude of the first head-impulse response. Then, the first peak position corrector 151 may multiply the first head-impulse response by the calculated ratio. This aligns the amplitude of the first head-impulse response with the amplitude of the first binaural room impulse response.
[0247] Fig. 23 is a diagram illustrating an example of an appropriate first angle according to the present embodiment. Fig. 24 is a diagram illustrating an example of an appropriate first delay time according to the present embodiment. Fig. 25 is a diagram illustrating an example of an appropriate first volume attenuation amount according to the present embodiment.
[0248] FIG. 23 shows the distance to the sound image position perceived by the subject (perceived distance), that is, how far away the sound was heard in a given direction, when the first angle was changed from 0 degrees to 180 degrees. The larger the perceived distance, the stronger the out-of-head feeling, and it can be said that three-dimensional sound is effectively perceived. Here, the conditions set are that the first volume attenuation amount (volume attenuation amount β) is -3 dB and the first delay time is 2.2 ms. As shown in FIG. 23, a high out-of-head feeling is obtained by setting the first direction to a direction of 105 degrees or 120 degrees.
[0249] FIG. 24 also shows the perceived distance perceived by the test subject when the first delay time was varied from 0 ms to 3.4 ms. Here, the conditions were set as follows: first volume attenuation (volume attenuation β) was -3 dB, and the first angle was 105 degrees. As shown in FIG. 24, a high out-of-head feeling was obtained by setting the first delay time from 2.4 ms to 2.8 ms, and a sufficient out-of-head feeling was obtained by setting the first delay time from 1.8 ms to 3.0 ms. However, since an increase in delay time leads to a deterioration in sound quality, a relatively short first delay time is appropriate. Therefore, it is recommended to set the first delay time from 1.8 ms to 2.4 ms, for example, 2.2 ms.
[0250] FIG. 25 also shows the perceived distance perceived by the test subject when the first volume attenuation amount (volume attenuation amount β) was varied from -30 dB to 0 dB. Note that here, the conditions were set as follows: the first delay time was 2.2 ms, and the first angle was 105 degrees. As shown in FIG. 25, a high out-of-head feeling was obtained by setting the first volume attenuation amount to -5 dB to -3 dB, and no further improvement in the out-of-head feeling was observed even when the first volume attenuation amount was set to -3 dB or more. Note that, because loud reflected sounds can cause deterioration in sound quality, it is considered best to keep the volume attenuation amount as small as possible.
[0251] (Variation) Next, a description will be given of a sound processing device according to a modification of the embodiment described above. In the modification described below, the configuration that is substantially the same as that of the embodiment described above will be omitted by referring to the description above.
[0252] In the above-described embodiment, the direct sound component of the binaural room impulse response is replaced with the direct sound component of the head impulse response, thereby combining the direct sound component of the head impulse response and the reverberation sound component of the binaural room impulse response. In contrast, in the first modification of the embodiment, the head impulse response is convolved with the binaural room impulse response in which the amplitudes of the direct sound components have all been replaced with 0, thereby combining the direct sound component of the head impulse response and the reverberation sound component of the binaural room impulse response.
[0253] FIG. 26 is a block diagram showing the configuration of the first creating unit 15 in the first modification of the present embodiment.
[0254] The first creating unit 15 shown in FIG. 26 includes a first specifying unit 161, a first amplitude substituting unit 162, a first amplitude setting unit 163, and a first convolution unit 164.
[0255] The first determination unit 161 determines a replacement time at which the amplitude of the first binaural room impulse response is closest to zero in the terminal period of the direct sound component of the first binaural room impulse response. The terminal period is, for example, a period from 1.5 msec to 2.0 msec. The terminal period may be, for example, a period from 2.0 msec to 2.5 msec. The terminal period may also be any period between 1.5 msec and 2.5 msec. The propagation time of the direct sound may be calculated based on the positions of the sound source and the microphone in the measurement environment, and the terminal period may be determined based on the calculated propagation time. Depending on the sampling time, the amplitude may not necessarily be zero. Therefore, the first determination unit 161 determines a replacement time at which the amplitude of the first binaural room impulse response is closest to zero.
[0256] The first amplitude substitution unit 162 substitutes all amplitudes of the direct sound components of the first binaural room impulse response with 0. That is, the first amplitude substitution unit 162 substitutes all amplitudes of the first binaural room impulse response before the substitution time identified by the first identification unit 161 with 0.
[0257] The first amplitude setting unit 163 sets the amplitude at time 0 of the first binaural room impulse response, in which the amplitudes of all direct sound components have been replaced with 0 by the first amplitude replacing unit 162, to 1. By setting the amplitude at time 0 of the first binaural room impulse response to 1, when the first head-based impulse response is convolved with the first binaural room impulse response, the direct sound component of the first head-based impulse response appears in the first binaural room impulse response.
[0258] The first convolution unit 164 convolves the first head-based impulse response with the first binaural room impulse response, the amplitude of which is set to 1 at time 0 by the first amplitude setting unit 163. This combines the direct sound component of the first head-based impulse response and the reverberation sound component of the first binaural room impulse response.
[0259] FIG. 27 is a block diagram showing the configuration of the second creation unit 25 in the first modification of the present embodiment.
[0260] The second creation unit 25 shown in FIG. 27 includes a second identification unit 261, a second amplitude substitution unit 262, a second amplitude setting unit 263, and a second convolution unit 264.
[0261] The second identification unit 261 identifies a replacement time at which the amplitude of the second binaural room impulse response is closest to zero during the terminal period of the direct sound component of the second binaural room impulse response. The terminal period is, for example, a period from 1.5 msec to 2.0 msec. The terminal period may be, for example, a period from 2.0 msec to 2.5 msec. The terminal period may also be any period between 1.5 msec and 2.5 msec. The propagation time of the direct sound may be calculated based on the positions of the sound source and the microphone in the measurement environment, and the terminal period may be determined based on the calculated propagation time. Depending on the sampling time, the amplitude may not necessarily be zero. Therefore, the second identification unit 261 identifies a replacement time at which the amplitude of the second binaural room impulse response is closest to zero.
[0262] The second amplitude substitution unit 262 substitutes all amplitudes of the direct sound components of the second binaural room impulse response with 0. That is, the second amplitude substitution unit 262 substitutes all amplitudes of the second binaural room impulse response before the substitution time identified by the second identification unit 261 with 0.
[0263] The second amplitude setting unit 263 sets the amplitude at time 0 of the second binaural room impulse response, in which the amplitudes of all direct sound components have been replaced with 0 by the second amplitude replacing unit 262, to 1. By setting the amplitude at time 0 of the second binaural room impulse response to 1, when the second head-based impulse response is convolved with the second binaural room impulse response, the direct sound component of the second head-based impulse response appears in the second binaural room impulse response.
[0264] The second convolution unit 264 convolves the second head-based impulse response with the second binaural room impulse response, whose amplitude at time 0 is set to 1 by the second amplitude setting unit 263. This combines the direct sound component of the second head-based impulse response and the reverberation sound component of the second binaural room impulse response.
[0265] Next, a description will be given of the first synthesized binaural room impulse response creation process in Modification 1 of the present embodiment. Note that the second synthesized binaural room impulse response creation process is the same as the first synthesized binaural room impulse response creation process, so only the first synthesized binaural room impulse response creation process will be described below.
[0266] FIG. 28 is a flowchart illustrating a first synthesized binaural room impulse response creation process by the first creation unit 15 according to the first modification of the embodiment of the present disclosure.
[0267] First, in step S41, the first identifying unit 161 identifies a replacement time at which the amplitude of the first binaural room impulse response is closest to zero in the terminal period of the direct sound component of the first binaural room impulse response.
[0268] Next, in step S42, the first amplitude replacing unit 162 replaces all amplitudes of the direct sound components of the first binaural room impulse response before the replacement time identified by the first identifying unit 161 with zero.
[0269] Next, in step S43, the first amplitude setting unit 163 sets the amplitude at time 0 of the first binaural room impulse response, in which the amplitudes of all direct sound components have been replaced with 0 by the first amplitude replacing unit 162, to 1.
[0270] Next, in step S44, the first convolution unit 164 convolves the first head-dependent impulse response with the first binaural room impulse response, the amplitude of which at time 0 is set to 1 by the first amplitude setting unit 163.
[0271] The above first synthesized binaural room impulse response creation process creates a first synthesized binaural room impulse response by combining the direct sound component of the first head impulse response and the reverberation sound component of the first binaural room impulse response. Also, the second synthesized binaural room impulse response creation process creates a second synthesized binaural room impulse response by combining the direct sound component of the second head impulse response and the reverberation sound component of the second binaural room impulse response.
[0272] In the above-described first synthesized binaural room impulse response creation process, a first synthesized binaural room impulse response may be created for each of the left channel and the right channel. Also, a first synthesized binaural room impulse response may be created for each of a plurality of head angles. Furthermore, when the angle of the front direction of the user's head relative to the sound arrival direction is acquired, a first synthesized binaural room impulse response corresponding to the acquired angle may be created.
[0273] In this way, by convolving the first head-impulse response with the first binaural room impulse response in which the amplitudes of all direct sound components are replaced with 0 and the amplitude at time 0 is set to 1, it is possible to synthesize the direct sound components of the first head-impulse response and the reverberation sound components of the first binaural room impulse response. Also, by convolving the second head-impulse response with the second binaural room impulse response in which the amplitudes of all direct sound components are replaced with 0 and the amplitude at time 0 is set to 1, it is possible to synthesize the direct sound components of the second head-impulse response and the reverberation sound components of the second binaural room impulse response.
[0274] FIG. 29 is a diagram showing an example of a first binaural room impulse response in the first modification of the present embodiment.
[0275] Note that, in the following, only the first binaural room impulse response, the first head-based impulse response, and the first synthesized binaural room impulse response will be described, but the second binaural room impulse response, the second head-based impulse response, and the second synthesized binaural room impulse response are similar to the first binaural room impulse response, the first head-based impulse response, and the first synthesized binaural room impulse response.
[0276] In Fig. 29, the vertical axis represents amplitude, the horizontal axis represents time (msec), and the solid lines represent first binaural room impulse responses 221L and 221R. The waveform diagram on the left side of Fig. 29 represents first binaural room impulse response 221L of the left channel when the predetermined direction (0 degrees) from which sound arrives coincides with the front direction of the user, and the waveform diagram on the right side of Fig. 29 represents first binaural room impulse response 221R of the right channel when the predetermined direction (0 degrees) from which sound arrives coincides with the front direction of the user.
[0277] As shown in FIG. 29, the first identification unit 161 identifies replacement times 223L, 223R at which the amplitudes of the first binaural room impulse responses 221L, 221R are closest to 0 in the terminal period (e.g., 1.5 msec to 2.0 msec) of the direct sound components of the first binaural room impulse responses 221L, 221R.
[0278] FIG. 30 is a diagram illustrating the process of replacing all amplitudes of direct sound components of the first binaural room impulse response with 0 in the first modification of the present embodiment.
[0279] In Fig. 30, the vertical axis represents amplitude, the horizontal axis represents time (msec), and the solid lines represent first binaural room impulse responses 221L and 221R. The waveform diagram on the left side of Fig. 30 represents first binaural room impulse response 221L of the left channel when the predetermined direction (0 degrees) from which sound arrives coincides with the front direction of the user, and the waveform diagram on the right side of Fig. 30 represents first binaural room impulse response 221R of the right channel when the predetermined direction (0 degrees) from which sound arrives coincides with the front direction of the user.
[0280] As shown in FIG. 30, the first amplitude substitution unit 162 substitutes, with zero, all amplitudes of the direct sound components of the first binaural room impulse responses 221L and 221R before substitution times 223L and 223R.
[0281] FIG. 31 is a diagram illustrating the process of setting the amplitude of the first binaural room impulse response at time 0 to 1 in the first modification of the present embodiment.
[0282] In Fig. 31, the vertical axis represents amplitude, the horizontal axis represents time (msec), and the solid lines represent first binaural room impulse responses 221L and 221R. The waveform diagram on the left side of Fig. 31 represents first binaural room impulse response 221L of the left channel when the predetermined direction (0 degrees) from which sound arrives coincides with the front direction of the user, and the waveform diagram on the right side of Fig. 31 represents first binaural room impulse response 221R of the right channel when the predetermined direction (0 degrees) from which sound arrives coincides with the front direction of the user.
[0283] As shown in FIG. 31, the first amplitude setting unit 163 sets the amplitudes of the first binaural room impulse responses 221L and 221R, in which the amplitudes of the direct sound components have all been replaced by 0 by the first amplitude replacing unit 162, to 1 at time 0.
[0284] FIG. 32 is a diagram showing an example of a first synthesized binaural room impulse response in the first modification of the present embodiment.
[0285] In Fig. 32, the vertical axis represents amplitude, the horizontal axis represents time (msec), and the solid lines represent first synthesized binaural room impulse responses 224L and 224R. The waveform diagram on the left side of Fig. 32 represents first synthesized binaural room impulse response 224L of the left channel when the predetermined direction (0 degrees) from which sound arrives coincides with the front direction of the user, and the waveform diagram on the right side of Fig. 32 represents first synthesized binaural room impulse response 224R of the right channel when the predetermined direction (0 degrees) from which sound arrives coincides with the front direction of the user.
[0286] The first synthesized binaural room impulse responses 224L and 224R are created by convolving the first head-based impulse response with the first binaural room impulse responses 221L and 221R, in which the amplitudes of all direct sound components are replaced with 0 and the amplitude at time 0 is set to 1. Therefore, the first synthesized binaural room impulse responses 224L and 224R are composed of the direct sound components of the first head-based impulse response and the reverberation sound components of the first binaural room impulse responses 221L and 221R.
[0287] Next, a sound reproducing device according to a second modification of the present embodiment will be described.
[0288] Fig. 33 is a block diagram showing the functional configuration of a sound reproduction device 100A according to Modification 2 of this embodiment. Fig. 34 is a block diagram showing the detailed functional configuration of a sound processing device 101A according to Modification 2 of this embodiment. As shown in Figs. 33 and 34, the sound reproduction device 100A according to Modification 2 includes a sound processing device 101A. The sound processing device 101A differs in configuration from the sound processing device 101 according to the above embodiment in that it includes a third processing unit 141.
[0289] The third processing unit 141 generates a third output sound signal of the second reflected sound. The third processing unit 141 generates the third output sound signal by convolving a third synthesized binaural room impulse response for localizing the sound included in the sound information including the start sound as a sound that arrives from a second direction having a second angle with respect to the direction of the sound image of the start sound that is greater than 0 degrees and less than 360 degrees and different from the first angle, and that has a second delay time that is greater than 0 and different from the first delay time, and a second volume attenuation amount that is greater than 0 and different from the first volume attenuation amount, with respect to the reproduced sound perceived by the first output sound signal.
[0290] In addition, the third processing unit 141 generates a third output sound signal by convolving a third synthesized binaural room impulse response for sound information including a reproduced sound, in order to localize the sound included in the information as a sound that arrives from a second direction having a second angle with respect to the predetermined direction that is greater than 0 degrees and less than 360 degrees and different from the first angle, and has a second delay time that is greater than 0 and different from the first delay time, and a second volume attenuation amount that is greater than 0 and different from the first volume attenuation amount, with respect to the reproduced sound perceived by the first output sound signal.
[0291] The third synthesized binaural room impulse response is created by synthesizing a direct sound component of a third head-based impulse response that exhibits the first head characteristic and is associated with a third sound image angle formed by the second direction and the front direction of the user, and a reverberant sound component of a third binaural room impulse response that exhibits the second head characteristic and spatial characteristic and is associated with the third sound image angle. The third head-based impulse response and the third binaural room impulse response are measured using different heads.
[0292] In addition, the third synthesized binaural room impulse response is created by synthesizing a direct sound component of the third head impulse response that indicates the first head characteristic and that corresponds to a third rotation angle formed by the second direction and the front direction of the user, and a reverberation sound component of the third binaural room impulse response that indicates the second head characteristic and spatial characteristic and that corresponds to the third rotation angle.
[0293] The third processing unit 141 convolves a third synthesized binaural room impulse response for localizing the sound in a second direction with respect to the sound information of the input start sound, and outputs a third output sound signal with attenuated volume via a third volume attenuation unit 146. The third output sound signal is input to a third equalizer (EQ) 142. The sound processing device 101A further includes a third EQ 142. The third EQ 142 adjusts low- and high-frequency sounds of the third output sound signal, and outputs the third output sound signal to the combiner 150.
[0294] Furthermore, the third processing unit 141 convolves a third synthesized binaural room impulse response for localizing the sound in a second direction with respect to the sound information of the input playback sound, and outputs a third output sound signal with attenuated volume via a third volume attenuation unit 146. The third output sound signal is input to a third equalizer (EQ) 142.
[0295] The sound processing device 101A further includes a second angle determination unit 140. The second angle determination unit 140 determines a preset second angle. The second angle determination unit 140 also calculates a third sound image angle formed by a second direction having a second angle determined with respect to the direction of the sound image of the start sound and a front direction of the user obtained by the acquisition unit 111, and outputs the calculated third sound image angle to the third signal processing unit 143. The second angle determination unit 140 also calculates a third rotation angle formed by the second direction having a second angle determined with respect to a predetermined direction and a front direction of the user obtained by the acquisition unit 111, and outputs the calculated third rotation angle to the third signal processing unit 143.
[0296] As the sound image of the start sound moves, the third sound image angle changes. Even if the sound image of the start sound moves, the magnitude of the second angle remains unchanged. Therefore, as the sound image of the start sound moves, the position of the sound image of the second reflected sound also moves. For example, if the angle between the front direction of the user and the direction of the sound image of the start sound is 270 degrees and the second angle is 250 degrees, the angle between the front direction of the user and the second direction is 160 degrees. Then, if the angle between the front direction of the user and the direction of the sound image of the start sound changes to 300 degrees, the angle between the front direction of the user and the first direction changes to 190 degrees.
[0297] The third processing unit 141 includes a third signal processing unit 143 , a third combined BRIR convolution unit 144 , a second delay unit 145 , and a third volume attenuation unit 146 .
[0298] The third signal processing unit 143 generates a third synthesized binaural room impulse response by synthesizing the direct sound component of the third head impulse response and the reverberation sound component of the third binaural room impulse response. The configuration of the third signal processing unit 143 is the same as the configuration of the second signal processing unit 133 shown in Fig. 12, and the configuration of the third creation unit (not shown) included in the third signal processing unit 143 is the same as the configuration of the second creation unit 25 shown in Fig. 15 or 27.
[0299] The third synthesis BRIR convolution unit 144 generates a third output sound signal by convolving the third synthesis binaural room impulse response with sound information including the start sound. The third synthesis BRIR convolution unit 144 also generates a third output sound signal by convolving the third synthesis binaural room impulse response with sound information including the playback sound.
[0300] The second delay unit 145 delays the third output sound signal according to the second delay time.
[0301] The third volume attenuation unit 146 attenuates the volume of the third output sound signal. The third volume attenuation unit 146 attenuates the volume of the third output sound signal by the volume attenuation amount γ (second volume attenuation amount).
[0302] The combiner 150 outputs to the driver 104 an output sound signal obtained by combining the first output sound signal generated by the first processing unit 121, the second output sound signal generated by the second processing unit 131, and the third output sound signal generated by the third processing unit 141. The combiner 150 outputs an output sound signal obtained by adding the first output sound signal, the second output sound signal, and the third output sound signal together. That is, in the sound processing device 101A, the second processing unit 131 and the third processing unit 141 each generate two different reflected sounds, and the combiner 150 superimposes these on the reproduced sound. When two reflected sounds are generated and superimposed on the reproduced sound as in the present second modification, it is possible to further improve the effect of expanding into a three-dimensional sound field depending on the conditions.
[0303] When the output sound signal output from the combiner 150 in the above manner is reproduced by the driver 104, a sound field such as that shown in FIG. 35 is formed.
[0304] Fig. 35 is a diagram for explaining the arrival direction of a sound output by a sound processing device 101A according to Modification 2 of the present embodiment. Fig. 35 shows a plan view of a virtual three-dimensional sound field from the same viewpoint as Fig. 10.
[0305] As shown in FIG. 35, the first reflected sound is localized in a direction that forms a first angle clockwise from the predetermined direction (position P2). The second reflected sound is localized in a direction that forms a second angle clockwise from the predetermined direction (position P3). As shown in FIG. 35, the first angle and the second angle do not match, and the directions are not line-symmetric with respect to a two-dot chain line that is parallel to the front and rear of the user 99 (also parallel to the predetermined direction). If the first direction and the second direction were line-symmetric, depending on the conditions, the two reflected sounds may be superimposed and localized as a single reflected sound behind the user 99. Therefore, the second angle is an angle such that the difference angle obtained by subtracting the second angle from 360 degrees does not match the first angle.
[0306] 35, the first angle and the second angle are positioned behind the virtual boundary plane that separates the head of the user 99 into front and rear. Therefore, the first angle and the second angle are both set to angles within an angle range in which the angle with respect to the predetermined direction is greater than 90 degrees and less than 270 degrees.
[0307] Note that the second angle, second delay time, and second volume attenuation amount described above are values that are preset by the sound processing device 101A, or values that change depending on the sensing result by the sensor 103, similar to the first angle, first delay time, and first volume attenuation amount, but at least one of them may be configured to be adjustable by a value arbitrarily input by the user 99. In other words, the sound processing device 101A may accept an input from the user 99 to adjust at least one of the second angle, second delay time, and second volume attenuation amount.
[0308] Furthermore, in the above description, the second reflected sound of the reproduced sound is described, but the second reflected sound of the start sound is localized in a direction having a second angle clockwise from the direction of the sound image of the start sound. When the start sound is reproduced, the second angle is preferably set to an angle within an angle range from the direction of the sound image of the start sound that is greater than 90 degrees and less than 270 degrees. Furthermore, the second angle is set such that the difference angle obtained by subtracting the second angle from 360 degrees does not match the first angle.
[0309] Next, the operation of the above-described sound reproducing device 100A will be described with reference to FIG.
[0310] FIG. 36 is a flowchart showing the operation of the sound processing device 101A according to the second modification of the present embodiment.
[0311] First, the operations of steps S51 to S55 are carried out in the same manner as the operations of steps S1 to S5 described with reference to FIG.
[0312] Next, in step S56, the third processing unit 141 generates a third output sound signal by convolving the sound information including the start sound with a third synthesized binaural room impulse response for localizing the sound included in the information as a sound that arrives from a second direction and has a second delay time greater than 0 and a volume attenuation amount γ (second volume attenuation amount) greater than 0 with respect to the reproduced sound perceived by the first output sound signal.
[0313] The processing of steps S54, S55, and S56 may be performed in a different order, or may be performed in parallel.
[0314] Next, in step S57, the combiner 150 combines the first output sound signal generated by the first processing unit 121, the second output sound signal generated by the second processing unit 131, and the third output sound signal generated by the third processing unit 141, and outputs the combined output sound signal.
[0315] The process in step S58 is the same as the process in step S7 shown in FIG. 16, and therefore a description thereof will be omitted.
[0316] When the output sound signal output in this manner is reproduced by the driver 104, the two reflected sounds are superimposed on the starting sound, and are perceived as a three-dimensional sound by the user 99. In particular, since only two reflected sounds are generated, no large-scale computing device or the like is required in this case either, and effective stereophonic sound can be perceived by the user 99.
[0317] The sound processing device may further increase the number of processing sections to superimpose three or more reflected sounds onto the reproduced sound.
[0318] (Other embodiments) Although the embodiments have been described above, the present disclosure is not limited to the above-described embodiments.
[0319] For example, in the above embodiment, when a start sound is played, a first output sound signal and a second output sound signal are generated, and an output sound signal obtained by combining the generated first output sound signal and second output sound signal is output, but the present disclosure is not particularly limited to this. When a start sound is played, the second output sound signal may not be generated, and only the first output sound signal may be generated and only the first output sound signal may be output.
[0320] Furthermore, for example, although the above embodiment has been described with reference to an example in which sound does not follow the movement of the user's head, the contents of the present disclosure are also effective in cases in which sound follows the movement of the user's head. In other words, in an operation in which the user perceives a predetermined sound as arriving from a first position that moves relatively with the movement of the user's head, if the amount of variation in the arrival direction of the predetermined sound is smaller than a threshold, a stereophonic filter may be selected to emphasize the variation.
[0321] In the above embodiment, the first processing unit 121 generates the first output sound signal by convolving the first synthesized binaural room impulse response with the sound information including the start sound, and the second processing unit 131 generates the second output sound signal by convolving the second synthesized binaural room impulse response with the sound information including the start sound. However, the present disclosure is not limited to this. The first processing unit 121 may generate the first output sound signal by convolving the sound information including the start sound with a first head-related transfer function for localizing the sound as arriving from the direction of a sound image. The second processing unit 131 may generate the second output sound signal by convolving the sound information with a second head-related transfer function for localizing the sound as arriving from a first direction having a first angle with the direction of the sound image that is greater than 0 degrees and less than 360 degrees, and having a first delay time greater than 0 and a first volume attenuation greater than 0 with respect to the start sound perceived by the first output sound signal.
[0322] Furthermore, for example, the sound reproduction device described in the above embodiment may be realized as a single device including all of the components, or may be realized by allocating each function to multiple devices and coordinating these multiple devices. In the latter case, an information processing device such as a smartphone, a tablet terminal, or a personal computer may be used as the device corresponding to the processing module.
[0323] As a configuration different from that described in the above embodiment, for example, a decoding processing unit can correct the original sound information to select a modified stereophonic filter. Specifically, the decoding processing unit in this example is a processing unit that generates information about a predetermined direction included in the sound information and corrects the original sound information. The decoding processing unit calculates the angle of variation in the predetermined direction on the time axis, and when the calculated angle of variation in the predetermined direction is smaller than a threshold, corrects the information about the predetermined direction so that the predetermined sound is perceived by the user as being more emphasized than when the angle of variation in the predetermined direction is equal to or greater than the threshold. This allows the modified stereophonic filter in the above embodiment to be applied simply by selecting a stereophonic filter that defines the arrival direction of the predetermined sound based on the corrected information about the predetermined direction output from the decoding processing unit.
[0324] In this way, the information processing method and the like of the present disclosure may be realized by correcting information relating to a predetermined direction in the original sound information. For example, by simply inserting the above-described decoding processing unit into a processing unit that performs decoding processing in a conventional stereophonic sound reproduction device, it is possible to realize a sound reproduction device that can achieve the same effects as those disclosed in the present disclosure.
[0325] Note that some or all of the functions of the device according to the embodiment of the present disclosure may be realized by a processor such as a CPU executing a program.
[0326] Furthermore, all the numbers used above are merely examples to specifically explain the present disclosure, and the present disclosure is not limited to the numbers used as examples.
[0327] The order in which the steps are executed shown in the above flowchart is merely an example for specifically explaining the present disclosure, and other orders may be used as long as similar effects are obtained. Also, some of the steps may be executed simultaneously (in parallel) with other steps. [Industrial Applicability]
[0328] The technology according to the present disclosure can clearly convey to the user the effect of out-of-head localization of a sound image, and is therefore useful as a technology for localizing a sound image outside the user's head. [Explanation of symbols]
[0329] 11 First HRIR Database 12 1st BRIR Database 13 1st HRIR Acquisition Department 14 1st BRIR Acquisition Department 15 First Creation Section 16 First output section 21 Second HRIR Database 22 Second BRIR Database 23 2nd HRIR Acquisition Department 24 2nd BRIR Acquisition Department 25 Second Creation Department 26 Second output section 99 users 100,100A sound reproduction device 101,101A Sound Processing Device 102 Communication Module 103 Sensors 104 Driver 105 Start reception desk 111 Acquisition Department 112 Start sound playback section 113 Sound image control section 120 Dereverberation processing unit 121 First Processing Section 122 1st EQ 123 First signal processing section 124 First synthetic BRIR convolution section 125 First volume attenuation section 130 First angle determination unit 131 Second Processing Section 132 2nd EQ 133 Second signal processing section 134 Second synthetic BRIR convolution section 135 First Delay Section 136 Second volume attenuation section 140 Second angle determination unit 141 Third Processing Section 142 3rd EQ 143 Third signal processing section 144 Third synthetic BRIR convolution section 145 Second Delay Section 146 Third volume attenuation section 150 Combiner 151 First peak position correction unit 152 1st Specific Part 153 1st direct sound component replacement section 154 First sound pressure level change unit 161 1st Specific Part 162 First amplitude substitution unit 163 First amplitude setting unit 164 First convolution section 251 Second peak position correction unit 252 Second Specific Part 253 2nd direct sound component replacement section 254 Second sound pressure level change unit 261 Second Specific Part 262 Second Amplitude Substitution Unit 263 Second amplitude setting unit 264 Second Convolution Unit
Claims
1. 1. A computer-implemented method for audio processing, comprising the steps of: receiving an instruction to start an acoustic processing for causing a user to perceive a reproduced sound as a sound arriving from a predetermined direction in a three-dimensional sound field; A sound image of a start sound is localized at a start position outside the head of the user, and the localized sound image is moved so as to pass to the side of the user; An acoustic processing method comprising:
2. the starting position is to the side of the user; The movement includes localizing the sound image at the start position and moving the sound image from the start position to a playback position in the predetermined direction to which the playback sound arrives, along the periphery of the user. The acoustic processing method according to claim 1.
3. the starting position is to the side of the user; The movement includes localizing the sound image at the start position and moving the sound image from the start position along the periphery of the user, passing behind the user, toward a playback position in the predetermined direction to which the playback sound arrives. The acoustic processing method according to claim 1.
4. the starting position is to the side of the user; The movement includes localizing the sound image at the start position, moving the sound image from the start position to a playback position that exists in the predetermined direction to which the playback sound arrives, along the periphery of the user, and further moving the sound image around the user from the playback position. The acoustic processing method according to claim 1.
5. the start position is a playback position that exists in the predetermined direction to which the playback sound arrives, The movement includes localizing the sound image at the start position and moving the sound image around the user from the start position. The acoustic processing method according to claim 1.
6. If the start instruction is accepted, playing the start sound. The acoustic processing method according to any one of claims 1 to 5.
7. The movement includes, when the start instruction is accepted, localizing the sound image of the playback sound being played back when the start instruction is accepted at the start position as the sound image of the start sound. The acoustic processing method according to any one of claims 1 to 5.
8. The movement is generating a first output sound signal by convolving a first head related transfer function for localizing a sound as a sound arriving from the direction of the sound image with respect to sound information including the start sound; generating a second output sound signal by convolving a second head related transfer function for localizing the sound as a sound arriving from a first direction having a first angle with respect to a direction of the sound image that is greater than 0 degrees and less than 360 degrees, and having a first delay time greater than 0 and a first volume attenuation greater than 0 with respect to the start sound perceived by the first output sound signal; outputting an output sound signal obtained by combining the generated first output sound signal and the generated second output sound signal; Including, The acoustic processing method according to any one of claims 1 to 5.
9. The movement is generating a first output sound signal by convolving a first synthetic binaural room impulse response for localizing a sound as arriving from the direction of the sound image with respect to sound information including the start sound; generating a second output sound signal by convolving a second synthetic binaural room impulse response for localizing the sound as a sound arriving from a first direction having a first angle with respect to a direction of the sound image that is greater than 0 degrees and less than 360 degrees, and having a first delay time greater than 0 and a first volume attenuation greater than 0 with respect to the start sound perceived by the first output sound signal; outputting an output sound signal obtained by combining the generated first output sound signal and the generated second output sound signal; Including, the first synthesized binaural room impulse response is created by synthesizing a direct sound component of a first head impulse response that indicates a first head characteristic of the user and that corresponds to a first sound image angle formed by a direction of the sound image and a front direction of the user, and a reverberation sound component of a first binaural room impulse response that indicates a second head characteristic different from the first head characteristic and a spatial characteristic of a space where the user is located and that corresponds to the first sound image angle, the second synthesized binaural room impulse response is created by synthesizing a direct sound component of a second head impulse response that indicates the first head characteristic and that is associated with a second sound image angle formed by the first direction and the front direction of the user, and a reverberation sound component of a second binaural room impulse response that indicates the second head characteristic and the spatial characteristic and that is associated with the second sound image angle. The acoustic processing method according to any one of claims 1 to 5.
10. The output sound signal is reproduced using headphones or earphones worn on the user's head. The acoustic processing method according to claim 9.
11. The first angle is an angle with respect to the direction of the sound image that is within an angle range of greater than 90 degrees and less than 270 degrees. The acoustic processing method according to claim 9.
12. The generation of the first output sound signal includes: trimming the waveform of the first binaural room impulse response and the waveform of the first head-based impulse response; identifying a replacement time at which a sound pressure difference between the first head-impulse response and the first binaural room impulse response becomes minimum in a terminal period of the direct sound component of the first head-impulse response; replacing the direct sound component of the first binaural room impulse response before the replacement time with the direct sound component of the first head-based impulse response before the replacement time; Including, The generation of the second output sound signal includes: trimming the waveform of the second binaural room impulse response and the waveform of the second head-based impulse response; identifying a replacement time at which a sound pressure difference between the second head-impulse response and the second binaural room impulse response becomes minimum in a terminal period of the direct sound component of the second head-impulse response; replacing the direct sound component of the second binaural room impulse response before the replacement time with the direct sound component of the second head-based impulse response before the replacement time; Including, The acoustic processing method according to claim 9.
13. the trimming of the first head-impulse response includes trimming a waveform of the first binaural room impulse response and a waveform of the first head-impulse response so that a time when the amplitude of the first binaural room impulse response is maximized coincides with a time when the amplitude of the first head-impulse response is maximized; The trimming of the second head-impulse response includes trimming a waveform of the second binaural room impulse response and a waveform of the second head-impulse response so that a time when the amplitude of the second binaural room impulse response is maximized coincides with a time when the amplitude of the second head-impulse response is maximized. The acoustic processing method according to claim 12.
14. The generation of the first output sound signal includes: Replacing all amplitudes of direct sound components of the first binaural room impulse response to zero; setting the amplitude of the first binaural room impulse response at time 0 to 1; convolving the first binaural room impulse response with the first head-based impulse response; Including, The generation of the second output sound signal includes: Replacing all amplitudes of direct sound components of the second binaural room impulse response to 0; setting the amplitude of the second binaural room impulse response at time 0 to 1; convolving the second binaural room impulse response with the second head-based impulse response; Including, The acoustic processing method according to claim 9.
15. A sound processing device including a processor, The processor, receiving an instruction to start acoustic processing for causing a user to perceive a reproduced sound as a sound arriving from a predetermined direction in a three-dimensional sound field; A sound image of the start sound is localized at a start position outside the head of the user, and the localized sound image is moved so as to pass to the side of the user. Sound processing equipment.
16. receiving an instruction to start acoustic processing for causing a user to perceive a reproduced sound as a sound arriving from a predetermined direction in a three-dimensional sound field; causing a computer to function so as to localize a sound image of a start sound at a start position outside the head of the user and to move the localized sound image so as to pass to the side of the user; Sound processing program.
Citation Information
Patent Citations
Acoustic processing apparatus, acoustic processing method, and program
WO2023106070A1
