Acoustic reproduction method, program, and acoustic reproduction system

Abstract

To provide an acoustic reproduction method and an acoustic reproduction system for causing a user to perceive stereophonic sounds through more appropriate calculation processing.SOLUTION: An acoustic reproduction method, in which a user 99 is caused to perceive a first sound as a sound arriving from a first position P1 in a three-dimensional sound field and the user 99 is caused to perceive a second sound as a sound arriving from a second position P2 different from the first position P1, includes an acquisition step for acquiring movement speed of the head of the user 99 and a generation step for generating an output sound signal for causing the user to perceive sounds arriving from predetermined positions in the three-dimensional sound field, where, in the generation step, if the movement speed acquired in the acquisition step is greater than a first threshold, the output sound signal is generated to cause the user 99 to perceive the first sound and the second sound as sounds arriving from a third position P3 between the first position P1 and the second position P2.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to a sound reproduction system and a sound reproduction method. [Background technology]

[0002] Conventionally, there is known a technology relating to sound reproduction that allows a user to perceive three-dimensional sound by controlling the position of a sound image, which is a sensory sound source object, in a virtual three-dimensional space (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-18620 Summary of the Invention [Problem to be solved by the invention]

[0004] However, generating sounds that allow the user to perceive a three-dimensional sound requires a huge amount of computational processing, and conventional sound reproduction methods have not always performed appropriate computational processing.

[0005] In view of the above, an object of the present disclosure is to provide a sound reproduction method and the like that allows a user to perceive three-dimensional sound through more appropriate calculation processing. [Means for solving the problem]

[0006] A sound reproduction method according to one embodiment of the present disclosure is a sound reproduction method that allows a user to perceive a first sound as a sound arriving from a first position in a three-dimensional sound field, and allows the user to perceive a second sound as a sound arriving from a second position different from the first position, and includes an acquisition step of acquiring the movement speed of the user's head, and a generation step of generating an output sound signal that allows the user to perceive the sound arriving from a predetermined position in the three-dimensional sound field, wherein in the generation step, when the acquired movement speed is greater than a first threshold, the output sound signal is generated that allows the user to perceive the first sound and the second sound as sounds arriving from a third position between the first position and the second position.

[0007] Furthermore, an audio reproduction system according to one embodiment of the present disclosure is an audio reproduction system that allows a user to perceive a first sound as a sound arriving from a first position in a three-dimensional sound field, and allows the user to perceive a second sound as a sound arriving from a second position different from the first position, and includes an acquisition unit that acquires the movement speed of the user's head, and a generation unit that generates an output sound signal that allows the user to perceive the sound arriving from a predetermined position in the three-dimensional sound field, and when the acquired movement speed is greater than a first threshold, the generation unit generates the output sound signal that allows the user to perceive the first sound and the second sound as sounds arriving from a third position between the first position and the second position.

[0008] Furthermore, one aspect of the present disclosure can also be realized as a program for causing a computer to execute the above-described sound reproduction method.

[0009] These comprehensive or specific aspects may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium. [Effects of the Invention]

[0010] According to the present disclosure, it is possible to allow a user to perceive stereoscopic sound through more appropriate calculation processing. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a schematic diagram showing a use example of a sound reproduction system according to an embodiment. [Figure 2] FIG. 2 is a block diagram showing a functional configuration of the sound reproduction system according to the embodiment. [Figure 3] FIG. 3 is a flowchart showing the operation of the sound reproduction system according to the embodiment. [Figure 4] FIG. 4 is a first diagram illustrating a third position where a sound image is localized by a third head-related transfer function according to the embodiment. [Figure 5] FIG. 5 is a flowchart showing the operation of the sound reproduction system according to the modified embodiment. [Figure 6A] FIG. 6A is a first diagram illustrating a third position where a sound image is localized by a third head-related transfer function according to a modification of the embodiment. [Figure 6B] FIG. 6B is a second diagram illustrating a third position where a sound image is localized by a third head-related transfer function according to a modification of the embodiment. [Figure 6C] FIG. 6C is a third diagram illustrating a third position where a sound image is localized by a third head-related transfer function according to a modification of the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] (Knowledge that served as the basis for disclosure) Conventionally, there has been known a technology relating to sound reproduction that allows a user to perceive three-dimensional sound by controlling the position of a sound image, which is a sound source object perceived by the user, in a virtual three-dimensional space (hereinafter sometimes referred to as a three-dimensional sound field) (see, for example, Patent Document 1). By localizing a sound image at a predetermined position in the virtual three-dimensional space, the user can perceive the sound as if it were emitted from the predetermined position. To localize a sound image at a predetermined position in the virtual three-dimensional space in this way, for example, calculation processing is required to generate a difference in sound arrival time between the two ears and a difference in sound level between the two ears, etc., for the collected sound so that the sound is perceived as three-dimensional sound.

[0013] As an example of such a calculation process, a process of convolving a head-related transfer function (HRTF) with a target sound signal to make the sound perceived as arriving from a predetermined position is known. By performing this HRTF convolution process at higher resolution, the sense of realism experienced by the user is improved. On the other hand, the HRTF convolution process is a relatively heavy calculation process and requires resources for the calculation. In other words, performing the HRTF convolution process at high resolution requires a high-performance calculation device and the power required for using the calculation device.

[0014] Furthermore, in recent years, there has been active development of technology related to virtual reality (VR). In virtual reality, the position in a virtual three-dimensional space does not follow the user's movements, and the focus is on allowing the user to experience as if they are moving within the virtual space. In particular, attempts have been made to enhance the sense of realism by incorporating auditory elements into visual elements in this virtual reality technology. For example, when a sound image is located in front of a user, if the user turns to the right, the sound image moves to the user's left, and if the user turns to the left, the sound image moves to the user's right. In this way, it becomes necessary to move the localized position of the sound image in the virtual space in the direction opposite to the user's movement.

[0015] In order to improve the sense of realism in a virtual space, it is necessary to increase the spatial resolution and perform convolution processing of head-related transfer functions. Therefore, in order to perform sound reproduction that allows a user to perceive highly realistic, three-dimensional sound, such as in the above-mentioned virtual reality, the constraints of the computing device and power consumption become more significant.

[0016] In view of the above, the present disclosure aims to provide a sound reproduction method and the like that allows a user to perceive three-dimensional sound through this appropriate calculation process by reducing the load of the calculation process while suppressing a decrease in the sense of realism.

[0017] More specifically, a sound reproduction method according to one embodiment of the present disclosure is a sound reproduction method that allows a user to perceive a first sound as a sound arriving from a first position in a three-dimensional sound field, and allows the user to perceive a second sound as a sound arriving from a second position different from the first position, and includes an acquisition step of acquiring the movement speed of the user's head, and a generation step of generating an output sound signal that allows the user to perceive the sound arriving from a predetermined position in the three-dimensional sound field, and in the generation step, if the acquired movement speed is greater than a first threshold, the output sound signal is generated that allows the user to perceive the first sound and the second sound as sounds arriving from a third position between the first position and the second position.

[0018] According to this sound reproduction method, a first sound perceived as arriving from a first position and a second sound perceived as arriving from a second position can be perceived as arriving from a third position when the user's head movement speed is greater than a first threshold. In this case, the process for localizing the sound image of the first sound at the first position and the process for localizing the sound image of the second sound at the second position can both be integrated into the process for localizing the sound image at the third position, thereby reducing the amount of processing. Furthermore, if the first threshold is set to a value that makes the user's perception of the sound image position unclear when the user's head movement speed exceeds the first threshold, the impact of changes in the sound image position on the sense of realism can be suppressed even when the above processing is performed. This also reduces the discomfort that the user may experience due to reduced processing load. Therefore, it is possible to allow the user to perceive three-dimensional sound through more appropriate calculation processing.

[0019] Furthermore, for example, in the generating step, when the acquired motion speed is equal to or less than the first threshold, a first head-related transfer function for localizing sound at the first position may be convolved with a first sound signal related to the first sound, and a second head-related transfer function for localizing sound at the second position may be convolved with a second sound signal related to the second sound, to generate the output sound signal; and when the acquired motion speed is greater than the first threshold, a third head-related transfer function for localizing sound at the third position may be convolved with a summed sound signal obtained by adding the second sound signal to the first sound signal, to generate the output sound signal.

[0020] When localizing the sound image of a first sound at a first position, a first head-related transfer function is convolved with a first sound signal related to the first sound, and when localizing the sound image of a second sound at a second position, a second head-related transfer function is convolved with a second sound signal related to the second sound. According to the above, when localizing the sound images of the first sound and the second sound at a third position, it is only necessary to perform a process of convolving the third head-related transfer function with a summed sound signal obtained by adding the first sound signal and the second sound signal to localize the sound at the third position. In other words, the process of convolving the first head-related transfer function with the first sound signal and the process of convolving the second head-related transfer function with the second sound signal can be shared by the process of convolving the third head-related transfer function with the summed sound signal. This reduces the amount of processing, making it possible to allow the user to perceive three-dimensional sound through more appropriate calculation processing.

[0021] Also, for example, the movement speed may be the rotation speed of the user's head around a first axis passing through the user's head, and the third position may be a position on a bisector that bisects the angle formed by the straight lines connecting the first position and the second position to the user within a virtual plane in which the three-dimensional sound field is viewed from the direction of the first axis.

[0022] According to this, the set third position can be used in accordance with the rotational movement of the user's head. In this case, the third position is set on a bisector that bisects the angle formed by the lines connecting the first position and the second position with the user in a virtual plane when the three-dimensional sound field is viewed from the direction of the first axis, which is the rotation axis. Therefore, the third position can be set in a direction between the directions of the first position and the second position as seen by the user, in accordance with the direction from which the sound is coming, which becomes ambiguous due to the rotational movement of the user. Therefore, it is possible to reduce the amount of processing while suppressing the sense of incongruity in the direction from which the sound is coming and allowing the user to perceive three-dimensional sound.

[0023] Furthermore, for example, the rotation speed may be acquired as the amount of rotation per unit time detected by a detector that moves integrally with the user's head and detects the amount of rotation around at least one of three mutually perpendicular axes as the rotation axis.

[0024] This allows the rotation speed of the user's head to be acquired as the movement speed using a detector, and therefore, based on the rotation speed acquired in the above manner, it is possible to suppress the sense of incongruity in the direction from which the sound is coming and allow the user to perceive three-dimensional sound.

[0025] Furthermore, for example, the movement speed may be the displacement speed of the user's head along a second axis direction passing through the user's head, and the displacement speed may be acquired as the displacement amount per unit time detected by a detector that moves integrally with the user's head and detects the displacement amount in at least one of three mutually perpendicular axes as the displacement direction.

[0026] The set third position can be used in accordance with the displacement of the user's head. At this time, the displacement speed of the user's head can be acquired using a detector. Therefore, based on the displacement speed acquired as described above, it is possible to suppress the sense of incongruity in the direction from which the sound is coming and allow the user to perceive three-dimensional sound.

[0027] Furthermore, for example, the sound reproduction method may cause the user to perceive a plurality of sounds arriving from each position within a predetermined area on the three-dimensional sound field, including the first position and the second position, the plurality of sounds including at least the first sound and the second sound, and in the generation step, when the movement speed is greater than the first threshold, the output sound signal may be generated to cause the user to perceive all of the plurality of sounds as sounds arriving from the third position.

[0028] This allows the user to perceive all of the multiple sounds within the predetermined range as sounds arriving from the third position. Therefore, the head-related transfer functions convolved with each sound within the predetermined range can be standardized by the head-related transfer function for localizing the sound image at the third position. This reduces the amount of processing required for convolution of the head-related transfer functions, and allows the user to perceive three-dimensional sound through more appropriate calculation processing.

[0029] Furthermore, for example, the sound reproducing method may cause a user to perceive a first intermediate sound as a sound arriving from a first intermediate position between the first position and the third position, and cause a user to perceive a second intermediate sound as a sound arriving from a second intermediate position between the second position and the third position, and the generating step may further generate the output sound signal for causing the user to perceive the first intermediate sound and the second intermediate sound as sounds arriving from the third position when the movement speed is equal to or less than the first threshold and greater than a second threshold that is smaller than the first threshold.

[0030] According to this, the same processing as described above can be applied within a narrow range including the first and second intermediate positions, which are closer to the third position than the first and second positions, respectively. Here, because the user's head movement speed is slower than the first threshold, consolidating sounds from the first and second positions, etc., at the third position would result in a perceived change in the sound image position, which could cause discomfort, and therefore this is not implemented. On the other hand, because the user's head movement speed is faster than the second threshold, consolidating sounds within a narrow range narrower than the predetermined range including the first and second positions, etc., at the third position would not result in a perceived change in the sound image position. Therefore, when the movement speed is equal to or less than the first threshold and greater than a second threshold that is smaller than the first threshold, sounds from the first and second intermediate positions within such a narrow range can be consolidated at the third position, thereby reducing the amount of computational processing. This makes it possible for the user to perceive three-dimensional sound through more appropriate computational processing.

[0031] Furthermore, an audio reproduction system according to one embodiment of the present disclosure is an audio reproduction system that allows a user to perceive a first sound as a sound arriving from a first position in a three-dimensional sound field, and allows the user to perceive a second sound as a sound arriving from a second position different from the first position, and includes an acquisition unit that acquires the movement speed of the user's head, and a generation unit that generates an output sound signal that allows the user to perceive the sound arriving from a predetermined position in the three-dimensional sound field, and when the acquired movement speed is greater than a first threshold, the generation unit generates the output sound signal that allows the user to perceive the first sound and the second sound as sounds arriving from a third position between the first position and the second position.

[0032] This makes it possible to realize a sound reproduction system that has the same effects as the sound reproduction method described above.

[0033] Furthermore, one aspect of the present disclosure can also be realized as a program for causing a computer to execute the above-described sound reproduction method.

[0034] This makes it possible to achieve the same effects as the above-described sound reproduction method using a computer.

[0035] Furthermore, these comprehensive or specific aspects may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0036] Hereinafter, embodiments will be described in detail with reference to the drawings. Note that the embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not recited in independent claims will be described as optional components. Note that each figure is a schematic diagram and is not necessarily an exact illustration. Furthermore, in each figure, substantially identical components are assigned the same reference numerals, and duplicated descriptions may be omitted or simplified.

[0037] In the following description, elements may be assigned ordinal numbers such as first, second, and third. These ordinal numbers are assigned to elements in order to identify them and do not necessarily correspond to a meaningful order. These ordinal numbers may be rearranged, newly added, or removed as appropriate.

[0038] (Embodiment) [overview] First, an overview of the sound reproduction system according to the embodiment will be described. Fig. 1 is a schematic diagram showing a use example of the sound reproduction system according to the embodiment. Fig. 1 shows a user 99 using the sound reproduction system 100.

[0039] The sound reproduction system 100 shown in FIG. 1 is used simultaneously with the stereoscopic video reproduction system 200. As described above, in this embodiment, by simultaneously viewing stereoscopic images and stereoscopic sound, the image enhances the auditory sense of realism, and the sound enhances the visual sense of realism, allowing the user to experience the image and sound as if they were actually at the scene where they were captured. For example, when an image (moving image) of people having a conversation is displayed, even if the localization of the sound image of the conversation sound is not aligned with the person's mouth, it is known that the user 99 perceives the conversation sound as coming from the person's mouth. In this way, the position of the sound image is corrected by visual information, and the sense of realism can be enhanced by combining the image and sound.

[0040] The stereoscopic video playback system 200 is an image display device worn on the head of the user 99. Therefore, the stereoscopic video playback system 200 moves integrally with the head of the user 99. For example, as shown in the figure, the stereoscopic video playback system 200 is a glasses-type device that is supported by the ears and nose of the user 99.

[0041] The stereoscopic video playback system 200 changes the displayed image in accordance with the movement of the user 99's head, thereby making the user 99 perceive the user as moving their head in the three-dimensional image space. In other words, when an object in the three-dimensional image space is located in front of the user 99, if the user 99 turns to the right, the object moves to the user 99's left, and if the user 99 turns to the left, the object moves to the user's right. In this way, the stereoscopic video playback system 200 moves the three-dimensional image space in the opposite direction to the user 99's movement.

[0042] The 3D video playback system 200 displays two images with a parallax difference to each of the left and right eyes of the user 99. The user 99 can perceive the three-dimensional position of an object on the image based on the parallax difference between the displayed images. Note that when the user 99 uses the sound playback system 100 with their eyes closed, for example, when using the system to play healing sounds for sleep induction, the 3D video playback system 200 does not need to be used at the same time. In other words, the 3D video playback system 200 is not an essential component of the present disclosure.

[0043] The sound reproduction system 100 is a sound presentation device worn on the head of the user 99. Therefore, the sound reproduction system 100 moves integrally with the head of the user 99. For example, the sound reproduction system 100 is two earplug-type devices worn independently on the left and right ears of the user 99. These two devices communicate with each other to present sounds for the right ear and the left ear in synchronization.

[0044] The sound reproduction system 100 changes the sound presented in accordance with the movement of the head of the user 99, thereby making the user 99 perceive as if he or she were moving his or her head within the three-dimensional sound field. For this reason, as described above, the sound reproduction system 100 moves the three-dimensional sound field in the opposite direction to the movement of the user 99.

[0045] It is known that when the movement of the user's 99's head exceeds a certain level, the user's ability to distinguish the position of a sound image in a three-dimensional sound field becomes unclear. The sound reproduction system 100 according to this embodiment reduces the load of calculation processing by utilizing this phenomenon. That is, the sound reproduction system 100 acquires the movement speed of the user's 99's head, and when the acquired movement speed is greater than a first threshold, causes the user to perceive multiple sounds that are perceived as arriving from within a predetermined area in the three-dimensional sound field as arriving from a single point within the predetermined area.

[0046] This predetermined area corresponds to a range where the user 99's perception of the sound image position becomes unclear due to a fast head movement speed. Therefore, it needs to be set for each user 99, and may be set, for example, by conducting an experiment in advance. Furthermore, since the predetermined area is also affected by the amount of head movement of the user 99, the amount of head movement of the user 99 may be detected, and the predetermined area may be set according to the amount of movement.

[0047] Similarly, the first threshold for the movement speed needs to be set to a value specific to the user 99, which indicates the movement speed at which the user 99's perception of the sound image position becomes unclear. Therefore, a value set in advance by conducting an experiment or the like may be adopted. Note that a generalized predetermined region and first threshold may be set by averaging the experimental results of multiple users 99.

[0048] [composition] Next, the configuration of the sound reproduction system 100 according to this embodiment will be described with reference to Fig. 2. Fig. 2 is a block diagram showing the functional configuration of the sound reproduction system according to this embodiment.

[0049] As shown in FIG. 2, the sound reproduction system 100 according to this embodiment includes a processing module 101, a communication module 102, a detector 103, and a driver 104.

[0050] The processing module 101 is an arithmetic device for performing various signal processing in the sound reproduction system 100. The processing module 101 includes, for example, a processor and a memory, and performs various functions by executing programs stored in the memory by the processor.

[0051] The processing module 101 includes an input unit 111, an acquisition unit 121, a generation unit 131, and an output unit 141. Details of each functional unit included in the processing module 101 will be described below together with details of other components of the processing module 101.

[0052] The communication module 102 is an interface device for accepting input of a sound signal to the sound reproduction system 100. The communication module 102 includes, for example, an antenna and a signal converter, and receives a sound signal from an external device via wireless communication. More specifically, the communication module 102 receives, using the antenna, a wireless signal representing a sound signal converted into a format for wireless communication, and reconverts the wireless signal into a sound signal using the signal converter. In this way, the sound reproduction system 100 acquires a sound signal from an external device via wireless communication. The sound signal acquired by the communication module 102 is input to the input unit 111. In this way, the sound signal is input to the processing module 101. Note that communication between the sound reproduction system 100 and the external device may be performed via wired communication.

[0053] The sound signal acquired by the sound reproduction system 100 is encoded in a predetermined format, such as MPEG-H Audio. As an example, the encoded sound signal includes information about the sound reproduced by the sound reproduction system 100 and information about the localization position when the sound image of the sound is localized at a predetermined position in a three-dimensional sound field. For example, the sound signal includes information about a plurality of sounds including a first sound and a second sound, and the sound images when each sound is reproduced are localized at different positions in the three-dimensional sound field.

[0054] This three-dimensional sound, together with the image viewed using the three-dimensional video reproduction system 200, can improve the sense of realism of the content being viewed. Note that the sound signal may contain only information about the sound. In this case, information about the localization position may be acquired separately. As described above, the sound signal includes a first sound signal related to a first sound and a second sound signal related to a second sound. However, sound images may be localized at different positions in a three-dimensional sound field by acquiring and simultaneously playing multiple sound signals each containing these separately. As such, there are no particular limitations on the form of the input sound signal, and the sound reproduction system 100 may be provided with an input unit 111 that can accommodate various forms of sound signals.

[0055] The detector 103 is a device for detecting the speed of movement of the head of the user 99. The detector 103 is configured by combining various sensors used for detecting movement, such as a gyro sensor and an acceleration sensor. In this embodiment, the detector 103 is built into the sound reproduction system 100. However, the detector 103 may be built into an external device, such as a 3D video reproduction system 200 that operates in response to the movement of the head of the user 99 in the same way as the sound reproduction system 100. In this case, the detector 103 does not need to be included in the sound reproduction system 100. Alternatively, the detector 103 may be an external imaging device or the like that captures an image of the head movement of the user 99 and detects the movement of the user 99 by processing the captured image.

[0056] The detector 103 is, for example, fixed integrally to the housing of the sound reproduction system 100 and detects the speed of movement of the housing. After the sound reproduction system 100 is worn by the user 99, it moves integrally with the head of the user 99, and as a result, the speed of movement of the head of the user 99 can be detected.

[0057] For example, the detector 103 may detect the amount of rotation about at least one of three mutually orthogonal axes in three-dimensional space as the rotation axis, or may detect the amount of displacement about at least one of the three axes as the displacement direction, as the amount of movement of the head of the user 99. Furthermore, the detector 103 may detect both the amount of rotation and the amount of displacement as the amount of movement of the head of the user 99.

[0058] The acquisition unit 121 acquires the movement speed of the head of the user 99 from the detector 103. More specifically, the acquisition unit 121 acquires the amount of movement of the head of the user 99 detected by the detector 103 per unit time as the movement speed. In this way, the acquisition unit 121 acquires at least one of the rotation speed and the displacement speed from the detector 103.

[0059] Here, the generation unit 131 determines whether or not the acquired movement speed of the head of the user 99 is greater than the first threshold value. Based on the result of this determination, the generation unit 131 decides whether or not to reduce the load of the calculation process. More detailed operations of the generation unit 131 will be described later. The generation unit 131 performs calculation processing on the input sound signal in accordance with the above-mentioned determination, and generates an output sound signal for presenting sound.

[0060] The output unit 141 is a functional unit that outputs the generated output sound signal to the driver 104. The driver 104 generates a waveform signal by performing signal conversion from a digital signal to an analog signal based on the output sound signal, generates sound waves based on the waveform signal, and presents the sound to the user 99. The driver 104 has, for example, a diaphragm and a drive mechanism such as a magnet and a voice coil. The driver 104 operates the drive mechanism in response to the waveform signal, causing the drive mechanism to vibrate the diaphragm. In this way, the driver 104 generates sound waves by vibrating the diaphragm in response to the output sound signal, and the sound waves propagate through the air to the ears of the user 99, causing the user 99 to perceive the sound.

[0061] [Operation] Next, the operation of the above-described sound reproduction system 100 will be described with reference to Fig. 3. Fig. 3 is a flowchart showing the operation of the sound reproduction system according to the embodiment. As shown in Fig. 3, first, when the operation of the sound reproduction system 100 starts, a first sound signal related to a first sound and a second sound signal related to a second sound are acquired (step S101). Here, a sound signal acquired by the communication module 102 from an external device is input to the input unit 111, and the processing module 101 acquires a sound signal including the first sound signal and the second sound signal.

[0062] Next, the acquisition unit 121 acquires the movement speed of the head of the user 99 as a detection result from the detector 103 (acquisition step S102). The generation unit 131 compares the acquired movement speed with a first threshold and determines whether the movement speed is greater than the first threshold (step S103). If the movement speed is equal to or less than the first threshold (No in step S103), the sound reproduction system 100 causes the user 99 to perceive the first sound and the second sound as sounds arriving from the first position and the second position, which are the original sound image positions, respectively. For this purpose, the generation unit 131 convolves the first sound signal with a first head-related transfer function for localizing the sound image at the first position. Furthermore, the generation unit 131 convolves the second sound signal with a second head-related transfer function for localizing the sound image at the second position (step S104). The generation unit 131 generates an output sound signal including the first sound signal and the second sound signal that have been subjected to the convolution process in this manner (step S105).

[0063] On the other hand, if the speed of movement is greater than the first threshold (Yes in step S103), the sound reproduction system 100 causes the user 99 to perceive the first sound and the second sound as arriving from a third position between the first and second positions, which are the original sound image positions of the first and second sounds. For this reason, the generation unit 131 adds the first sound signal and the second sound signal to generate an added sound signal related to the sound in which the first sound and the second sound are superimposed. Note that "between the first position and the second position" refers to, for example, the area sandwiched between an imaginary line passing through the first position and another imaginary line parallel to the first imaginary line and passing through the second position. In this case, the imaginary line and the other imaginary line may be included in the area.

[0064] The generation unit 131 further convolves this added sound signal with a third head-related transfer function for localizing a sound image at a third position (step S107). The generation unit 131 generates an output sound signal including the added sound signal that has been subjected to the convolution process in this manner (step S108). Note that steps S103 to S108 are collectively referred to as a generation step.

[0065] The output unit 141 outputs the output sound signal generated by the generation unit 131 to the driver 104, thereby driving the driver 104 and presenting a sound based on the output sound signal (step S106). In this way, the first sound and the second sound can be perceived together as sounds arriving from a third position. This simplifies the calculation process for localizing the sound image compared to when the first sound is perceived as a sound arriving from the first position and the second sound is perceived as a sound arriving from the second position. This temporarily reduces the required processing power, reducing heat generated by the processor and power consumption associated with the calculation process. Furthermore, as described above, the simplified calculation process also makes the user 99's perception of the sound image position unclear, so the impact on the sense of realism is minimal. In this way, the sound reproduction system 100 can simplify the calculation process as needed, allowing the user to perceive three-dimensional sound through more appropriate calculation process.

[0066] The third position explained above will now be described in more detail with reference to Fig. 4. Fig. 4 is a diagram illustrating the third position where a sound image is localized by a third head-related transfer function according to an embodiment. In Fig. 4, the sound image position in the three-dimensional sound field is indicated by a black dot, and the direction from which the sound arrives to the user 99 is indicated by an arrow extending from the black dot toward the user 99. The black dot indicating the sound image position also shows a virtual speaker.

[0067] In the example shown in FIG. 4, the user 99 is rotating his / her head, and the rotation speed of this rotation is greater than a first threshold. Note that when the user 99 displaces his / her head and the displacement speed of this displacement is greater than the first threshold, the following operation may be performed. In this example, as shown by the outlined double-headed arrow, the head of the user 99 is rotating around a first axis perpendicular to the paper surface. At this time, as shown in the figure, the third position P3 or P3a in this example is a position on the bisector indicated by the dotted arrow in the figure, which bisects the angle formed by the line connecting the first position P1 or P1a and the user 99 and the line connecting the second position P2 or P2a and the user 99.

[0068] In this way, by simplifying the calculation process for convolution of the head-related transfer functions, it is possible to allow the user 99 to perceive three-dimensional sound through more appropriate calculation process. Note that if the head-related transfer functions include information on the distance at which a sound image is localized, a configuration may be adopted in which a plurality of head-related transfer functions that localize sound images at a plurality of distances in the same sound arrival direction are prepared, and one head-related transfer function selected from these is convolved. In this case, since the arrival directions of the first sound and the second sound and the distances to the sound image positions are averaged, which can easily cause the user 99 to feel uncomfortable, a configuration for reducing the sense of discomfort, such as setting a smaller predetermined area, may be further included.

[0069] When the user 99 displaces his / her head, the description will be given assuming that the displacement speed of this displacement is greater than the first threshold value. In this example, for example, the user 99's head displaces along a second axis in the up-down direction along the paper surface. At this time, the third position P3 in this example is a position on an equidistant line that is perpendicular to the second axis direction and is equidistant from the first position P1 and the second position P2. By localizing the sound image at such a position, it is possible to set an average third position P3 in a distance range where discrimination becomes ambiguous in accordance with the displacement of the user 99's head. Note that the displacement direction of the user 99's head may be unidirectional.

[0070] Furthermore, when setting the third position, a position corresponding to either the first position or the second position itself may be set. For example, if the first sound is a person's dialogue in the content and the second sound is an environmental sound in the content, the first sound is given priority, and the sound image position set for the first sound is set as the third position. In this way, the first sound and the second sound are perceived as sounds arriving from the first position set as the third position. In this case, the first head-related transfer function for causing the user 99 to perceive the sounds as arriving from the first position is used as is.

[0071] In other words, in this example, since the head-related transfer function that has already been used is used, for example, as shown in the above example, it is not necessary to set a position that does not correspond to any of the sound image positions, such as the first position and the second position, originally set by the sound signal as the third position. In other words, the sound image position originally set by the sound signal can be set as the third position. Therefore, since the head-related transfer function for localizing a sound image at the originally set sound image position can be used, it is not necessary to use mapping information that maps the head-related transfer function for making the user 99 perceive sound as sound arriving from an arbitrary point in a three-dimensional sound field. Therefore, the process of determining the head-related transfer function for the set third position is simplified, and it is possible to make the user 99 perceive three-dimensional sound through more appropriate calculation processing. In this way, between the first position and the second position means a range that includes the first position and the second position themselves.

[0072] Furthermore, the third position may be set to a midpoint on a line segment spatially connecting the first position and the second position, or may simply be set to a random position between the first position and the second position.

[0073] [Variations] The operation of the sound reproduction system according to the modified example of this embodiment will be described below with reference to Fig. 5 and Fig. 6A to Fig. 6C. In the following description of the modified example of this embodiment, differences from the above embodiment will be mainly described, and substantially equivalent aspects will be omitted or simplified.

[0074] Fig. 5 is a flowchart showing the operation of a sound reproduction system according to a modified example of the embodiment. Fig. 6A is a first diagram illustrating a third position at which a sound image is localized by a third head-related transfer function according to a modified example of the embodiment. Fig. 6B is a second diagram illustrating a third position at which a sound image is localized by a third head-related transfer function according to a modified example of the embodiment. Fig. 6C is a third diagram illustrating a third position at which a sound image is localized by a third head-related transfer function according to a modified example of the embodiment. The sound reproduction system according to this modified example differs from the sound reproduction system 100 according to the above-described embodiment in that the sound to which the head-related transfer function is convolved with respect to the sound signal changes at the boundaries between the first threshold and the second threshold.

[0075] More specifically, in the sound reproduction system according to this modification, a second threshold value smaller than the first threshold value is set. As in the above embodiment, the first threshold value is used to determine whether or not to apply a third head-related transfer function that causes the user 99 to perceive the first sound and the second sound as sounds arriving from a third position. In this modification, the second threshold value is further used to make a determination to convolve a third head-related transfer function that causes the user 99 to perceive the first intermediate sound and the second intermediate sound, which are localized at a first intermediate position and a second intermediate position closer to the third position than the first sound and the second sound, as sounds arriving from the third position, thereby realizing a reduction in the amount of calculation processing.

[0076] Here, a determination is made based on the movement speed of the head of the user 99, and when the movement speed is equal to or less than a second threshold, the first sound is localized to the first position P1, the second sound is localized to the second position P2, the first intermediate sound is localized to the first intermediate position P1m (see FIG. 6A, etc.), and the second intermediate sound is localized to the second intermediate position P2m (see FIG. 6A, etc.). On the other hand, when the movement speed of the head of the user 99 is greater than the first threshold, as described above, a process is applied in which the third head related transfer function is convolved with the sound signals related to the first and second sounds (i.e., the first sound signal and the second sound signal). At this time, the third head related transfer function is also convolved with the sound signals related to the first and second intermediate sounds (i.e., the first intermediate sound signal and the second intermediate sound signal), and the first sound, the second sound, the first intermediate sound, and the second intermediate sound are all localized to the third position P3.

[0077] In addition, in this modification, when the speed of movement of the head of the user 99 is greater than the second threshold and equal to or less than the first threshold, the first sound is localized to the first position P1, the second sound is localized to the second position P2, and the first intermediate sound and the second intermediate sound are localized to the third position P3. That is, in this modification, when the speed of movement of the head of the user 99 is not so fast, such as equal to or less than the second threshold, the calculation process of convolution of the head related transfer functions is simplified for a smaller predetermined region (i.e., a narrow region) that does not include the first position P1 and the second position P2 but includes the first intermediate position P1m and the second intermediate position P2m.

[0078] 5, in the operation of the sound reproduction system according to this modification, the acquisition unit 121 acquires the motion speed (step S102), and then the generation unit 131 determines whether the motion speed is greater than a second threshold value (step S201). If the motion speed is equal to or less than the second threshold value (No in step S201), the process proceeds to step S202, where, as in the above embodiment, an operation of convolving a head-related transfer function for localizing a sound image at a position where it should originally be localized is performed for each sound signal (step S202). That is, a first sound signal related to a first sound is convolved with a first head-related transfer function for localizing a sound image at a first position P1, a second sound signal related to a second sound is convolved with a second head-related transfer function for localizing a sound image at a second position P2, a first intermediate sound signal related to a first intermediate sound is convolved with a first intermediate head-related transfer function for localizing a sound image at a first intermediate position P1m, and a second intermediate sound signal related to a second intermediate sound is convolved with a second intermediate head-related transfer function for localizing a sound image at a second intermediate position P2m.

[0079] On the other hand, if the motion speed is greater than the second threshold (Yes in step S201), the generation unit 131 further determines whether the motion speed is greater than the first threshold (step S204). If the motion speed is equal to or less than the first threshold (No in step S204), the sound reproduction system 100 causes the user 99 to perceive the first intermediate sound and the second intermediate sound as sounds arriving from a third position. For this reason, the generation unit 131 convolves the third head-related transfer function with an added sound signal obtained by adding the first intermediate sound signal related to the first intermediate sound and the second intermediate sound signal related to the second intermediate sound (step S205). The generation unit 131 generates an output sound signal including the first sound signal and the second sound signal subjected to the convolution process in this way, and an added sound signal obtained by adding the first intermediate sound signal and the second intermediate sound signal (step S206). Thereafter, the process proceeds to step S106, and the same operation as in the above embodiment is performed.

[0080] On the other hand, if the motion speed is greater than the first threshold (Yes in step S204), the process proceeds to step S207, and a process of convolving the third head-related transfer function with the summed sound signal obtained by adding the first sound signal and the second sound signal is performed in the same manner as in the above embodiment. In this modification, the first intermediate sound signal and the second intermediate sound signal are also added to this summed sound signal, and the first sound, the second sound, the first intermediate sound, and the second intermediate sound are perceived by the user 99 as sounds arriving from the third position P3.

[0081] As a result of the above operations, in the sound reproduction system according to the modified example of the present embodiment, when the movement speed of the user 99 is equal to or less than the second threshold, the sound image shown in Fig. 6A is formed in the three-dimensional sound field. Note that Fig. 6A shows a view of the three-dimensional sound field as seen from the first axis direction, similar to Fig. 4. As shown in Fig. 6A, when the movement speed of the user 99 is equal to or less than the second threshold, the first sound, the second sound, the first intermediate sound, and the second intermediate sound are each perceived by the user 99 as sounds arriving from the original sound image positions.

[0082] Furthermore, in the sound reproduction system according to this modification, when the movement speed of the user 99 is equal to or less than the first threshold and greater than the second threshold, the sound image shown in Fig. 6B is formed in the three-dimensional sound field. Note that Fig. 6B shows the three-dimensional sound field as seen from the first axis direction, similar to Fig. 4.

[0083] 6B , when the motion speed of the user 99 is equal to or less than the first threshold and greater than the second threshold, the first intermediate sound, which is actually perceived by the user 99 as arriving from a first intermediate position P1m that is closer to the third position P3 than the first position P1, is perceived by the user 99 as arriving from the third position P3. Similarly, when the motion speed is equal to or less than the first threshold and greater than the second threshold, the second intermediate sound, which is actually perceived by the user 99 as arriving from a second intermediate position P2m that is closer to the third position P3 than the second position P2, is perceived by the user 99 as arriving from the third position P3.

[0084] Furthermore, in the sound reproduction system according to this modification, when the movement speed of the user 99 is greater than the first threshold, the sound image shown in Fig. 6C is formed in the three-dimensional sound field. Note that Fig. 6C shows the three-dimensional sound field as seen from the first axis direction, similar to Fig. 4.

[0085] As shown in Figure 6C, when the movement speed of the user 99 is greater than the first threshold, all sounds that are originally localized at sound image positions included in a specified area that includes the first position P1 and the second position P2, including the first intermediate position P1m and the second intermediate position P2m, are perceived by the user 99 as sounds arriving from the third position P3.

[0086] By doing this, when the movement speed exceeds the second threshold, the user 99 perceives sound within a predetermined area whose size gradually corresponds to the movement speed of the user 99 as sound arriving from the third position P3. For example, in the figure, when the movement speed exceeds the first threshold, the user 99 perceives sound within the predetermined area indicated by the long dashed line as sound arriving from the third position P3. Furthermore, when the movement speed exceeds the second threshold but is equal to or less than the first threshold, the user 99 perceives sound within a small predetermined area (i.e., a narrow area) indicated by the dashed line as sound arriving from the third position P3.

[0087] At this time, the first intermediate position P1m and the second intermediate position P2m are considered as the third position P3. That is, the third position P3 is set based on the four positions of the first position P1, the second position P2, the first intermediate position P1m, and the second intermediate position P2m. Here, for example, the third position P3 is set to a position on a line connecting the user 99 with the center between the first position P1, the second position P2, the first intermediate position P1m, and the second intermediate position P2m, and at a distance equal to the shortest distance from each of the first position P1, the second position P2, the first intermediate position P1m, and the second intermediate position P2m to the position of the user 99. Alternatively, the third position P3 may be set to an average coordinate of coordinates corresponding to the four positions in a plane coordinate system viewed from the first axis direction.

[0088] Furthermore, three or more stages such as a third threshold value for the movement speed of the user 99 may be set, and a sound within a smaller predetermined area may be perceived by the user 99 as a sound arriving from a third position P3. There is no particular limit to the number of stages in the relationship between the movement speed and the size of the predetermined area.

[0089] Furthermore, the second threshold value may be set based on a numerical value specific to the user 99, such as the first threshold value described in the above embodiment, which indicates the movement speed at which the user 99's perception of the sound image position becomes unclear, or a generalized numerical value may be set.

[0090] (Other embodiments) Although the embodiments have been described above, the present disclosure is not limited to the above-described embodiments.

[0091] For example, although the above embodiment has been described with reference to an example in which sound does not follow the movement of the user's head, the contents of the present disclosure are also effective in cases in which sound does follow the movement of the user's head. That is, in an operation in which the user perceives a first sound as arriving from a first position that moves relatively with the movement of the user's head, and a second sound as arriving from a second position that moves relatively with the movement of the user's head, if the speed of head movement is greater than a first threshold, the first sound and the second sound are perceived as arriving from a third position that moves relatively with the movement of the user's head.

[0092] Even in this case, a process of convolving the head-related transfer functions for localizing the first sound and the second sound at the first position and the second position with each sound signal is performed, and the head-related transfer functions convolved with the sound signals are common across the first threshold, simplifying the calculation process. That is, as in the above embodiment, the required processing power is temporarily reduced, making it possible to reduce heat generated by driving the processor and power consumption associated with the calculation process. On the other hand, even if such a simplification of the calculation process is performed, if the user's head movement speed is high, it becomes difficult to accurately perceive the position of the sound image, and therefore the user is less likely to feel uncomfortable with the position of the sound image. Therefore, it is possible to allow the user to perceive three-dimensional sound through more appropriate calculation process.

[0093] Furthermore, for example, the sound reproduction system described in the above embodiment may be realized as a single device including all of the components, or may be realized by allocating functions to multiple devices and coordinating these multiple devices. In the latter case, an information processing device such as a smartphone, a tablet terminal, or a PC may be used as a device corresponding to the processing module.

[0094] The sound reproduction system of the present disclosure can also be realized as a sound processing device that is connected to a reproduction device having only a driver and that simply outputs to the reproduction device an output sound signal that has been subjected to convolution processing of a head-related transfer function based on an acquired sound signal. In this case, the sound processing device may be realized as hardware having a dedicated circuit, or as software that causes a general-purpose processor to execute specific processing.

[0095] In the above-described embodiment, the processing performed by a specific processing unit may be performed by another processing unit. The order of multiple processing operations may be changed, or multiple processing operations may be performed in parallel.

[0096] In the above embodiments, each component may be realized by executing a software program suitable for that component, or by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0097] Furthermore, each component may be realized by hardware. For example, each component may be a circuit (or integrated circuit). These circuits may form a single circuit as a whole, or each may be a separate circuit. Furthermore, each of these circuits may be a general-purpose circuit or a dedicated circuit.

[0098] Furthermore, the general or specific aspects of the present disclosure may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, etc. Furthermore, the general or specific aspects of the present disclosure may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0099] For example, the present disclosure may be realized as an audio signal reproducing method executed by a computer, or as a program for causing a computer to execute the audio signal reproducing method. The present disclosure may also be realized as a computer-readable non-transitory recording medium on which such a program is recorded.

[0100] In addition, this disclosure also includes forms obtained by applying various modifications to each embodiment that a person skilled in the art would think of, or forms realized by arbitrarily combining the components and functions of each embodiment within the scope that does not deviate from the intent of this disclosure. [Industrial Applicability]

[0101] The present disclosure is useful in reproducing sound that allows a user to perceive a three-dimensional sound accompanied by the movement of the user's head. [Explanation of symbols]

[0102] 99 users 100 Sound reproduction system 101 Processing Module 102 Communication Module 103 Detector 104 Driver 111 Input section 121 Acquisition Department 131 Generation part 141 Output section 200 3D video playback system P1, P1a 1st position P2, P2a 2nd position P3, P3a 3rd position P1m 1st intermediate position P2m 2nd intermediate position

Claims

1. 1. A method of sound reproduction performed by a sound reproduction system, comprising: acquiring a first sound signal linked to first position information indicating a first position in a three-dimensional sound field and a second sound signal linked to second position information indicating a second position different from the first position; an acquisition step of acquiring a movement speed of the user's head; and determining, when the acquired motion speed is equal to or less than a first threshold, that the first sound based on the first sound signal and the second sound based on the second sound signal are to be perceived by the user as sounds arriving from the first position and the second position, respectively. Sound reproduction method.

2. If the acquired motion speed is greater than the first threshold, a calculation process for localizing the sound is simplified. The sound reproducing method according to claim 1 .

3. If the acquired motion speed is greater than the first threshold, a common head-related transfer function is used to convolve the first sound signal and the second sound signal. The sound reproducing method according to claim 1 .

4. If the acquired motion speed is greater than the first threshold, an added sound signal is generated by adding the second sound signal to the first sound signal. The sound reproducing method according to claim 1 .

5. a generating step of generating an output sound signal for causing the user to perceive a sound arriving from a predetermined position in the three-dimensional sound field; In the generating step, when the acquired motion speed is equal to or less than the first threshold, a first head-related transfer function for localizing the sound at the first position is convolved with a first sound signal related to the first sound, and a second head-related transfer function for localizing the sound at the second position is convolved with a second sound signal related to the second sound, thereby generating the output sound signal that allows the user to perceive the first sound arriving from the first position and the second sound arriving from the second position. The sound reproducing method according to claim 1 .

6. a generating step of generating an output sound signal for causing the user to perceive a sound arriving from a predetermined position in the three-dimensional sound field; In the generating step, when the acquired motion speed is greater than the first threshold, a third head related transfer function for localizing the sound at a third position between the first position and the second position is convolved with an added sound signal obtained by adding the second sound signal to the first sound signal, thereby generating the output sound signal that allows the user to perceive the first sound and the second sound arriving from the third position. The sound reproducing method according to any one of claims 1 to 5.

7. The third position is the same as the first position or the second position. The sound reproducing method according to claim 6.

8. The third position is different from the first position and the second position. The sound reproducing method according to claim 6.

9. The third position is located in a region between an imaginary line passing through the first position and another imaginary line parallel to the first imaginary line and passing through the second position. The sound reproducing method according to claim 6.

10. The movement velocity is a rotation velocity of the user's head about a first axis passing through the user's head. The sound reproducing method according to claim 1 .

11. The third position is a position on a bisector that bisects an angle formed by lines connecting the first position and the second position with the user in a virtual plane in which the three-dimensional sound field is viewed from the direction of the first axis. The sound reproducing method according to claim 10.

12. The rotation speed is acquired as an amount of rotation per unit time detected by a detector that moves integrally with the user's head and detects an amount of rotation about at least one of three mutually orthogonal axes as a rotation axis. The sound reproducing method according to claim 11.

13. the movement velocity is a displacement velocity of the user's head along a second axis direction passing through the user's head; The displacement velocity is acquired as a displacement amount per unit time detected by a detector that moves integrally with the user's head and detects a displacement amount in a displacement direction along at least one of three mutually orthogonal axes. The sound reproducing method according to any one of claims 1 to 10.

14. the sound reproducing method includes making the user perceive a plurality of sounds that arrive from positions within a predetermined region in the three-dimensional sound field, the plurality of sounds including at least the first sound and the second sound, the plurality of sounds including at least the first sound and the second sound; determining that, when the speed of the movement is greater than the first threshold, all of the plurality of sounds are to be perceived by the user as sounds arriving from a third position between the first position and the second position; The sound reproducing method according to any one of claims 1 to 13.

15. The sound reproducing method includes: making a user perceive a first sound as a sound arriving from a first position; making the user perceive the second sound as a sound arriving from the second position; making the user perceive a first intermediate sound as a sound arriving from a first intermediate position between the first position and the third position, the first intermediate position being closer to the third position than the first position; and making the user perceive a second intermediate sound as a sound arriving from a second intermediate position between the second position and the third position, the second intermediate position being closer to the third position than the second position; determining that, when the speed of the movement is greater than the first threshold, the first sound, the second sound, the first intermediate sound, and the second intermediate sound are to be perceived by the user as sounds arriving from a third position; When the speed of the movement is equal to or less than the first threshold and greater than a second threshold that is smaller than the first threshold, it is determined that the first sound is to be perceived by the user as a sound arriving from the first position, the second sound is to be perceived by the user as a sound arriving from the second position, and the first intermediate sound and the second intermediate sound are to be perceived by the user as sounds arriving from the third position. The sound reproducing method according to any one of claims 1 to 14.

16. A method for causing a computer to execute the sound reproducing method according to any one of claims 1 to 15. program.

17. an input unit that acquires a first sound signal linked to first position information indicating a first position in a three-dimensional sound field and a second sound signal linked to second position information indicating a second position different from the first position; an acquisition unit that acquires a movement speed of the user's head; a generator that determines, when the acquired motion speed is equal to or less than a first threshold, to cause the user to perceive a first sound based on the first sound signal and a second sound based on the second sound signal as sounds arriving from the first position and the second position, respectively. Sound reproduction system.

Images