Sound field auxiliary method, sound field auxiliary device, and storage medium

Abstract

The present application relates to a sound field assistance method and a sound field assistance apparatus which can compare a sound of a virtual sound source and a simulated playback sound played in an object space. The sound field assistance method selects either position information of a sound source set in a virtual space or positioning information of the sound source when a sound of the sound source is simulated by an output sound from a speaker set in an object space, and adjusts a sound image positioning of the sound source realized by the speaker using a sound based on the selected position information and the positioning information.

Description

TECHNICAL FIELD

[0001] One embodiment of the present application relates to a sound field assisting method and a sound field assisting apparatus that perform a process for simulating a sound field realized by a sound source set in a virtual space in an object space in which a speaker is arranged. BACKGROUND

[0002] There are various techniques for simulating a sound of a sound source set in a virtual space in an actual space.

[0003] For example, a simulation system as shown in Patent Literature 1 sets positions of a plurality of virtual speakers in such a manner that the positions follow in order to maintain a relative positional relationship with a viewer on a virtual space in association with a change in the position of the viewer. Also, the simulation system as shown in Patent Literature 1 sets a volume balance of the plurality of virtual speakers.

[0004] The simulation system as shown in Patent Literature 1 performs an audio processing using the plurality of virtual speakers on the basis of the above setting.

[0005] Patent Literature 1: Japanese Patent Application Laid-Open No. 2017-184174

[0006] However, in a case where a sound set using a virtual sound source (virtual speaker of Patent Literature 1) is played back in an object space, the sound is played back by a speaker arranged in the object space and assigned with the virtual sound source. That is, the sound played back in the object space is a sound obtained by simulating a sound of the virtual sound source by a sound of the speaker arranged in the object space.

[0007] Also, in the past, a sound from the virtual sound source and a sound (simulated playback sound) played back by the speaker in the object space by simulation could not be compared. Therefore, the viewer could not confirm to what extent the sound from the virtual sound source could be simulated by the simulated playback sound, and could not easily perform adjustment. SUMMARY

[0008] Therefore, an object of one embodiment of the present application is to enable comparison of a sound of a virtual sound source and a simulated playback sound.

[0009] The sound field assisting method selects either position information of a sound source set on a virtual space or positioning information of the sound source when a sound of the sound source is simulated by an output sound from a speaker set in an object space, and adjusts a sound image positioning of the sound source realized by the speaker using a sound based on the selected position information and the positioning information.

[0010] EFFECT OF THE INVENTION

[0011] The sound field assisting method enables the viewer to compare the sound of the virtual sound source and the simulated playback sound. Attached Figure Description

[0012] Figure 1 This is a functional block diagram showing the structure of a sound field assisting system that includes the sound field assisting device according to the first embodiment of the present invention.

[0013] Figure 2 This is a diagram illustrating an example of the positional relationship between the sound source, the audiovisual point, and multiple loudspeakers in the sound field assistance method according to the first embodiment of the present invention, showing the position coordinates of the sound source, the audiovisual point, and the multiple loudspeakers.

[0014] Figure 3 It is a diagram showing the general situation of sound being emitted from a sound source, and a diagram showing the general situation of sound being reproduced and emitted by a loudspeaker.

[0015] Figure 4 This is a flowchart illustrating the first method of the sound field assisting method according to the first embodiment of the present invention.

[0016] Figure 5 This is a flowchart illustrating the second method of the sound field assisting method according to the first embodiment of the present invention.

[0017] Figure 6 This is a diagram illustrating an example of a GUI used for parameter adjustment.

[0018] Figure 7 This is a functional block diagram showing the structure of a sound field assisting system that includes the sound field assisting device according to the second embodiment of the present invention.

[0019] Figure 8 This is a diagram illustrating an example of the positional relationship between the sound source, the audiovisual point, multiple loudspeakers, and the virtual space in the sound field assistance method according to the second embodiment of the present invention.

[0020] Figure 9 This is an example diagram of a GUI used to represent the expansion and adjustment of sound positioning.

[0021] Figure 10 This is a flowchart illustrating the sound field assist method according to the second embodiment of the present invention.

[0022] Figure 11 This is a functional block diagram showing the structure of a sound field assisting system that includes the sound field assisting device according to the third embodiment of the present invention.

[0023] Figure 12 This is a flowchart illustrating the sound field assist method according to the third embodiment of the present invention. Detailed Implementation

[0024] Referring to the accompanying drawings, the sound field assisting method and sound field assisting device according to the embodiments of the present invention will be described.

[0025] In this embodiment, the object space is the space in which a viewer or listener actually hears the sound of a sound source set in a virtual space using a speaker or the like. More specifically, in the sound field assisting method of this embodiment, the object space does not represent the space where a speaker is actually placed, but rather a predetermined space where a viewer or listener hears the sound from that speaker. The virtual space is the space in the object space containing the sound source that is to be simulated.

[0026] [First Implementation]

[0027] Figure 1 This is a functional block diagram showing the structure of a sound field assisting system that includes the sound field assisting device according to the first embodiment of the present invention. Figure 2 (A) is a diagram illustrating an example of the positional relationship between the sound source, the viewing / listening point, and the multiple loudspeakers in the sound field assistance method according to the first embodiment of the present invention. Figure 2 (B) means Figure 2 (A) is a diagram showing the coordinates of the sound source, the audiovisual point, and the multiple loudspeakers. Figure 3 (A) is a diagram showing the general situation of sound emission from the sound source. Figure 3 (B) is a diagram showing the general situation of sound source being reproduced and played by loudspeaker.

[0028] like Figure 2 As shown in (A), an audiovisual point 900 for the viewer to watch and listen, and multiple speakers SP1-SP5 are configured in the object space 90. A virtual space is set in the object space 90. A sound source OBJ is set in the virtual space.

[0029] Furthermore, in this embodiment, there is one sound source, but there can be multiple sound sources. When there are multiple sound sources, the sound field assistance method described below can be applied to each of the multiple sound sources separately. Alternatively, the sound field assistance method described below can be applied to multiple sound sources collectively. In this embodiment, the case with one sound source is described. Also, in this embodiment, the number of loudspeakers is 5, but the number of loudspeakers is not limited to this.

[0030] The coordinate system of object space 90 and the coordinate system of virtual space can be set to have the same orthogonal three axes and center point. In this case, the position coordinates based on the coordinate system in object space 90 are the same as the position coordinates based on the coordinate system in virtual space. Furthermore, even if the coordinate system of object space 90 and the coordinate system of virtual space are not the same, in this case, it is only necessary to set the coordinate transformation matrix between object space 90 and virtual space.

[0031] like Figure 1 As shown, the sound field assist system includes a sound field assist device 10 and headphones 80. The sound field assist device 10 includes an audio-visual point setting unit 21, a sound source position setting unit 22, a speaker position setting unit 23, an adjustment operation unit 29, an analog playback sound signal generation unit 30, a selection unit 40, and a binaural processing unit 50. The sound field assist device 10 is implemented by a processing unit such as a CPU that executes the above-mentioned functional units, a storage medium that stores the program, and executes the program.

[0032] The audiovisual point setting unit 21 sets the position coordinates Pr of the audiovisual point 900 in the object space 90. The audiovisual point setting unit 21 outputs the position coordinates Pr of the audiovisual point 900 to the analog playback sound signal generation unit 30 and the binaural processing unit 50.

[0033] The sound source position setting unit 22 sets the position coordinates Pobj of the sound source OBJ in the virtual space (more specifically, the position coordinates of the sound source in the virtual space projected onto the object space 90). The sound source position setting unit 22 outputs the position coordinates Pobj of the sound source OBJ to the analog playback sound signal generation unit 30 and the binaural processing unit 50.

[0034] The speaker position setting unit 23 sets the position coordinates Psp1-Psp5 of the multiple speakers SP1-P5 in the object space 90. The speaker position setting unit 23 outputs the position coordinates Psp1-Psp5 of the multiple speakers SP1-P5 to the analog playback audio signal generation unit 30 and the binaural processing unit 50.

[0035] The adjustment operation unit 29 receives the operation input of adjustment parameters. The adjustment operation unit 29 outputs the adjustment parameters to the analog playback sound signal generation unit 30.

[0036] The analog playback sound signal generation unit 30 generates an analog playback sound signal for output to the speakers SP1-SP5 of the object space 90 based on the object playback sound signal.

[0037] Here, the object playback sound signal refers to the sound signal output from the sound source OBJ. The analog playback sound signal refers to the sound signal used for sound image localization of the sound source OBJ by a speaker that reproduces the sound source OBJ.

[0038] More specifically, the simulated playback sound signal generation section 30 calculates the positional relationship between the position coordinate Pobj of the sound source OBJ and the position coordinates Psp1-Psp5 of the plurality of speakers SP1-SP5, using the position coordinate Pr of the listening point 900 as a reference point. The simulated playback sound signal generation section 30 sets the sound image positioning information of the sound source OBJ using the positional relationship. The sound image positioning information is information set so that the sound source OBJ is played back at the listening point 900 by the sound output from the plurality of speakers SP1-SP5, and is information that determines the volume and output timing of the output sound from the plurality of speakers SP1-SP5.

[0039] The simulated playback sound signal generation section 30 sets the plurality of speakers that reproduce the sound source OBJ using the sound image positioning information of the sound source OBJ (refer to Figure 3 (B)). The simulated playback sound signal generation section 30 generates a simulated playback sound signal played by the plurality of speakers that reproduced the sound source OBJ. The simulated playback sound signal generation section 30 outputs the simulated playback sound signal to the selection section 40.

[0040] The selection section 40 receives an operation input from a listener or the like, and selects the object playback sound signal and the simulated playback sound signal. More specifically, if the setting (A) of listening to the sound directly output from the sound source OBJ set on the virtual space is selected, the selection section 40 selects and outputs the object playback sound signal. On the other hand, if the setting (B) of listening to the sound from the plurality of speakers reproduced is selected, the selection section 40 selects and outputs the simulated playback sound signal. In other words, if the position information of the sound source OBJ is selected, the object playback sound signal is selected and output, and if the positioning information of the sound source OBJ using the speakers is selected, the simulated playback sound signal is selected and output. Figure 3 Figure 3 (A) of the sound source OBJ is selected, the selection section 40 selects and outputs the object playback sound signal. On the other hand, if the setting (B) of listening to the sound from the plurality of speakers reproduced is selected, the selection section 40 selects and outputs the simulated playback sound signal. In other words, if the position information of the sound source OBJ is selected, the object playback sound signal is selected and output, and if the positioning information of the sound source OBJ using the speakers is selected, the simulated playback sound signal is selected and output.

[0041] The selection section 40 outputs the selected sound signal to the binaural processing section 50.

[0042] The binaural processing section 50 performs binaural processing on the sound signal selected by the selection section 40. In addition, the binaural processing uses a head transfer function, and the details are known, and a detailed description of the binaural processing is omitted.

[0043] ​More specifically, in a case where the selection section 40 selects the object playback sound signal, the binauralization processing section 50 performs binauralization processing on the sound signal of the sound source OBJ using the position coordinates Pobj of the sound source OBJ and the position coordinates Pr of the listening point 900. In a case where the selection section 40 selects the simulated playback sound signal, the binauralization processing section 50 performs binauralization processing on the simulated playback sound signal using the position coordinates Psp of the speaker SP that reproduced the sound source OBJ and the position coordinates Pr of the listening point 900.

[0044] For example, if it is Figure 2 (A), Figure 2 (B), Figure 3 (A), Figure 3 (B), in a case where the selection section 40 selects the object playback sound signal, the binauralization processing section 50 performs binauralization processing on the object playback sound signal using the position coordinates Pobj of the sound source OBJ and the position coordinates Pr of the listening point 900. In a case where the selection section 40 selects the simulated playback sound signal, the binauralization processing section 50 performs binauralization processing on the simulated playback sound signal using the position coordinates Psp1, Psp5 of the speakers SP1, SP5 that reproduced the sound source OBJ and the position coordinates Pr of the listening point 900.

[0045] The binauralization processing section 50 outputs the sound signal on which binauralization processing has been performed (binauralized signal) to the earphone 80.

[0046] The earphone 80 plays back the sound signal based on the binauralized signal. Furthermore, in the present embodiment, a manner in which the earphone 80 is used for playback is shown, but playback can also be performed using a 2-channel stereo speaker.

[0047] With the above-described structure, in a case where the object playback sound signal is selected, the listener is able to hear the sound of the sound source positioned at the position of the sound source OBJ (object playback sound) through the earphone 80. On the other hand, in a case where the simulated playback sound signal is selected, the listener is able to hear the sound of the sound source simulatedly positioned at the position of the sound source OBJ (simulated playback sound) through the earphone.

[0048] Thus, even without actually configuring a speaker in an actual space, the listener is able to comparatively hear the object playback sound and the simulated playback sound. Therefore, the listener is able to directly perceive the difference between the object playback sound and the simulated playback sound, and is able to judge whether the simulated playback sound is able to reproduce (simulate) the object playback sound with high precision, and whether there is a sense of discomfort between the object playback sound and the simulated playback sound.

[0049] Further, the listener can adjust the parameters for adjusting the simulated playback sound signal by referring to the above-mentioned perception result. Moreover, by repeatedly adjusting the parameters as mentioned above, the listener can reproduce the target playback sound with high precision through the simulated playback sound.

[0050] Further, here, a manner of adjusting the simulated playback sound signal for reproducing the sound of the sound source OBJ with high precision is shown. However, for example, in a case where the positions of the loudspeakers in the target space 90 are changed, the change of the parameters is difficult, but the position setting of the sound source OBJ can be changed, the above-mentioned sound after the binaural processing can be heard, and the listener changes the setting of the sound source OBJ to realize the desired sound field.

[0051] (Sound field assistance method 1)

[0052] Figure 4 is a flowchart showing the first sound field assistance method according to the first embodiment of the present application. Figure 4 The flow of the sound field assistance method shown in FIG. 1 is executed until the sound signal subjected to the binaural processing is output. Further, the sound field assistance method shown in FIG. 1 is executed until the sound signal subjected to the binaural processing is output. Figure 4 Detailed explanations of the respective processes shown in FIG. 1 are described above, and therefore, detailed explanations will be omitted below. Further, below, the case where the configuration manner shown in (A) and (B) is exemplified. Figure 2 (A), Figure 2 (B), Figure 3 (A), Figure 3 (B) is exemplified.

[0053] The sound source position setting section 22 sets the position of the sound source OBJ in the virtual space (S11). The loudspeaker position setting section 23 sets the positions of the loudspeakers SP1 to SP5 in the target space (S12).

[0054] The simulated playback sound signal generation section 30 reproduces the sound source OBJ at the loudspeakers SP1 and SP5 using the position coordinates Pobj of the sound source OBJ, the position coordinates Psp1 to Psp5 of the loudspeakers SP1 to SP5, and the position coordinates Pr of the listening point 900 (S13). The simulated playback sound signal generation section 30 generates the simulated playback sound signal using the reproduction result (S14).

[0055] The selection section 40 selects the object playback sound signal or the analog playback sound signal by operation from a viewer or the like (S15). For example, the sound field assist apparatus 10 has a GUI (Graphical User Interface) or the like. The GUI has an operation member for selecting a sound signal of a playback object. If the viewer selects output of the object playback sound signal, the selection section 40 selects the object playback sound signal (S150: YES). If the viewer selects output of the analog playback sound signal, the selection section 40 selects the analog playback sound signal (S150: NO). Further, the selection of the object playback sound signal and the analog playback sound signal can also be to set a switching time, and switching is automatically performed in accordance with the time.

[0056] The binauralization processing section 50 performs binauralization processing on the selected sound signal, and generates a binauralized signal. More specifically, if the object playback sound signal is selected, the binauralization processing section 50 performs binauralization processing on the object playback sound signal, and generates a binauralized signal of the object playback sound signal (S161). If the analog playback sound signal is selected, the binauralization processing section 50 performs binauralization processing on the analog playback sound signal, and generates a binauralized signal of the analog playback sound signal (S162).

[0057] The earphone 80 plays the binauralized signal (S17). More specifically, the earphone 80 plays the binauralized signal of the object playback sound signal if input thereto. The earphone 80 plays the binauralized signal of the analog playback sound signal if input thereto.

[0058] By performing the processing as described above, the sound field assist method can selectively provide the object playback sound and the analog playback sound to a viewer or the like.

[0059] (Sound field assist method 2)

[0060] Figure 5 is a flowchart showing a second method of the sound field assist method according to the first embodiment of the present application. Figure 5 The sound field assist method shown in Figure 4 The sound field assist method shown in Figure 5 The sound field assist method shown in Figure 4 The sound field assist method shown in Figure 2 (A), Figure 2 (B), Figure 3 (A), Figure 3 (B) is described below.

[0061] Figure 5 The sound field assist method shown in Figure 4The illustrated sound field auxiliary method performs the same processing up to step S17.

[0062] The listener switches the sound signal played by the processing of steps S15 to S17. As described above, the listener compares the sound of the binauralized signal of the object play sound signal and the sound of the binauralized signal of the simulated play sound signal.

[0063] If parameter adjustment is not needed (S23: NO), that is, the sound based on the binauralized signal of the object play sound signal can be reproduced with high precision based on the sound of the binauralized signal of the simulated play sound signal, the processing ends. If parameter adjustment is needed (S23: YES), the listener performs parameter adjustment using the adjustment operation section 29 (S24). The simulated play sound signal generation section 30 generates the simulated play sound signal using the adjusted parameters (S14).

[0064] Further, the adjusted parameters are, for example, settings of the reproduction of the sound source OBJ and the speaker, the volume level of the simulated play sound signal, the frequency characteristics, and the like. Figure 6 is a diagram showing one example of a GUI for parameter adjustment. As shown in Figure 6 The GUI 100 has a position relationship confirmation window 111, a waveform confirmation window 112, and a plurality of operation sections 113. The plurality of operation sections 113 each have a knob 1131 and an adjustment value display window 1132.

[0065] The position relationship confirmation window 111 displays the sound sources OBJ1 to OBJ3 and the plurality of speakers SP1 to SP5 at respectively set position coordinates. The setting of the speaker SP assigned to the sound source OBJ can be achieved, for example, by selecting the sound source OBJ and the speaker SP to be reproduced in the position relationship confirmation window 111.

[0066] The waveform confirmation window 112 displays the waveform of the simulated play sound signal. The selection of the displayed simulated play sound signal is switched, for example, by selecting the plurality of speakers SP1 to SP5 displayed in the position relationship confirmation window 111.

[0067] The plurality of operation sections 113 are, for example, operation sections that receive the settings of Q, filter processing, and the settings of gain values of the simulated play sound signal for a plurality of frequency bands (Hi, Mid, Low). The knob 1131 receives an operation from the listener, and the adjustment value display window 1132 displays the value set by the knob 1131. The parameters of the simulated play sound signal are adjusted by operation input based on the plurality of operation sections 113. Further, the waveform achieved based on the adjusted parameters is displayed in the waveform confirmation window 112.

[0068] The viewer can adjust and set the parameters while observing the GUI 100.

[0069] Thereafter, the viewer adjusts the parameters while comparing the sound of the binauralized signal based on the object playback sound signal and the sound involved in the binauralized signal based on the analog playback sound signal. As described above, the viewer can adjust so that the sound of the binauralized signal based on the analog playback sound signal can reproduce the sound of the binauralized signal based on the object playback sound signal with high precision, that is, the analog playback sound based on the loudspeaker can analog the object playback sound of the sound source OBJ with high precision. Further, the "adjustment section" of the present application is realized by the unit that outputs and compares the object playback sound and the analog playback sound and the adjustment operation section 29.

[0070] Further, the sound field assisting apparatus 10 and the sound field assisting method of the present embodiment show a manner of comparing the object playback sound and the analog playback sound involved in the binauralized playback. However, the sound field assisting apparatus 10 and the sound field assisting method of the present embodiment can also compare the waveform or the frequency spectrum of the object playback sound signal, the HOA (High Order Ambisonics) and the waveform or the frequency spectrum of the analog playback sound signal, the HOA (High Order Ambisonics) to adjust the parameters, for example.

[0071] [2nd Embodiment]

[0072] The sound field assisting apparatus and the sound field assisting method involved in the 2nd embodiment of the present application will be described with reference to the drawings.

[0073] Figure 7 is a functional block diagram showing the structure of a sound field assisting system including the sound field assisting apparatus involved in the 2nd embodiment of the present application. Figure 8 is a diagram showing one example of the positional relationship of the sound source, the listening point, the plurality of loudspeakers, and the virtual space of the sound field assisting method involved in the 2nd embodiment of the present application.

[0074] As shown in Figure 7 The sound field assisting apparatus 10A involved in the 2nd embodiment differs from the sound field assisting apparatus 10 involved in the 1st embodiment in that the reverb processing section 60 is added. The other structure of the sound field assisting apparatus 10A is the same as that of the sound field assisting apparatus 10, and the description of the same parts is omitted.

[0075] The sound field assisting apparatus 10A has the reverb processing section 60. The object playback sound signal and the analog playback sound signal are input to the reverb processing section 60.

[0076] The reverb processing section 60 generates the initial reflection sound signal and the reverberation sound signal using the information of the virtual space 99. The initial reflection sound signal is a sound signal simulating a sound of a sound source OBJ reaching the listening point after being reflected (once reflected) by a wall of the virtual space. The initial reflection sound signal is determined by the geometry of the virtual space, the position of the sound source OBJ of the virtual space, and the position of the listening point. The reverberation sound signal is a sound signal simulating a sound reaching the listening point after being reflected multiple times in the virtual space. The reverberation sound signal is determined by the geometry of the virtual space and the position of the listening point of the virtual space.

[0077] More specifically, the reverb processing section 60 generates the initial reflection sound signal and the reverberation sound signal for the object playback sound signal using the position information of the sound source OBJ, the information of the virtual space 99, and the position information of the listening point. The reverb processing section 60 outputs the generated initial reflection sound signal and the reverberation sound signal to the selection section 40 by attaching them to the object playback sound signal.

[0078] In addition, the reverb processing section 60 generates the initial reflection sound signal and the reverberation sound signal for the simulation playback sound signal using the position information of the sound source OBJ, the position information of the speakers SP1 to SP5, the information of the virtual space 99, and the position information of the listening point. As a specific example, the reverb processing section 60 sets a virtual sound source that virtually represents a position of occurrence of the initial reflection sound for the sound source OBJ in accordance with the position information of the sound source OBJ and the listening point and the information of the virtual space 99. The reverb processing section 60 generates the initial reflection sound signal in accordance with a positional relationship between the virtual sound source and the speaker SP to which the virtual sound source is assigned. The reverb processing section 60 generates the reverberation sound signal using the geometry of the virtual space and the position of the listening point of the virtual space. The reverb processing section 60 outputs the initial reflection sound signal and the reverberation sound signal generated in the above-described manner to the selection section 40 by attaching them to the simulation playback sound signal.

[0079] With the above-described configuration, the sound field assisting apparatus 10A is capable of attaching respective reverb components (initial reflection sound and reverberation sound) to the object playback sound (sound from the sound source OBJ) and the simulation playback sound (sound simulated by the speakers) and outputting them. As described above, the listener is also able to judge the accuracy of reproduction of the object playback sound based on the simulation playback sound while taking the reverb components into consideration.

[0080] Further, the reverb processing section 60 is also capable of imparting a sense of expansion and localization to the initial reflection sound signal and the reverberation sound signal for the simulation playback sound signal. In this case, the listener is able to perform adjustment, for example, using the GUI shown in FIG. 17. Figure 9 Figure 9 is a diagram showing one example of a GUI for adjustment of a sense of expansion and localization of sound. As shown in FIG. 17, the GUI includes a display area 1701 in which a sound field is displayed, and a display area 1702 in which a sound field is displayed. The display area 1701 is a display area in which a sound field is displayed in a state in which the sound field is not adjusted. The display area 1702 is a display area in which a sound field is displayed in a state in which the sound field is adjusted. The display area 1701 is a display area in which a sound field is displayed in a state in which the sound field is not adjusted. The display area 1702 is a display area in which a sound field is displayed in a state in which the sound field is adjusted. Figure 9 ​As shown, the GUI 100A has a setting display window 111A, an output state display window 115, and a plurality of operation members 116. The plurality of operation members 116 have a knob 1161, and an adjustment value display window 1162.

[0081] The setting display window 111A displays the virtual sound source SS, the plurality of speakers SP, the virtual space 99, and the audio point RP set with respect to the sound source OBJ at the respective set position coordinates.

[0082] The plurality of operation members 116 are operation members for setting a weight value and a shape (Shape) value. The weight is a weighting of a sound in a playback space toward a prescribed direction, and the weight value is a value that determines the weighting. The shape (Shape) is an expansion of a sound in a playback space toward a prescribed direction, and the shape value is a value that determines the expansion. The operation members 116 for setting the weight value have operation members for setting the weight of the left and right, the weight of the front and back, and the weight of the upper and lower, respectively, and have operation members for setting a gain value and operation members for setting a delay amount. The operation members 116 for setting the shape value have operation members for setting the expansion, and have operation members for setting a gain value and operation members for setting a delay amount. The listener can adjust the expansion and the localization of the sound by operating the plurality of operation members 116.

[0083] The output state display window 115 displays the expansion and the localization of the sound realized by the weight value and the shape value set by the plurality of operation members 116 in a graphical manner. As described above, the listener can easily recognize the expansion and the localization of the sound set by the plurality of operation members 116 as an image. In addition, the output state display window 115 can also display an image representing a head and an image representing the expansion and the localization of the sound in combination with the image of the head in a case where the sound after the binauralization processing is heard through the earphone 80.

[0084] As described above, the listener can also judge the accuracy of the reproduction of the object playback sound based on the simulated playback sound while considering the expansion and the localization of the sound.

[0085] In addition, the listener can also adjust the shape of the virtual space 99, the position with respect to the playback space, the position of the sound source OBJ, and the positions of the plurality of speakers SP, for example, by operating the setting display window 111A. In this case, the sound field assist device generates the object playback sound signal and the simulated playback sound signal in accordance with the various contents adjusted, and implements the same reverberation processing. As described above, the listener can also judge the accuracy of the reproduction of the object playback sound based on the simulated playback sound after the adjustment.

[0086] (Sound field assist method of the second embodiment)

[0087] Figure 10 is a flowchart showing a sound field assist method according to the second embodiment of the present application. Figure 10 The sound field assist method shown in Figure 4 The sound field assist method shown in Figure 10 The sound field assist method shown in Figure 4 The sound field assist method shown in

[0088] Figure 10 The sound field assist method shown in Figure 4 The sound field assist method shown in

[0089] The reverb processing section 60 generates reverb components (initial reflection sound signals and reverberation sound signals) for the object playback sound signals and the simulated playback sound signals and adds them to the object playback sound signals and the simulated playback sound signals (S31).

[0090] The sound field assist device 10A uses the object playback sound signals to which the reverb components are added and the simulated playback sound signals to which the reverb components are added and executes the process of step S15 and the subsequent processes.

[0091] Thus, the sound field assist method according to the second embodiment can output the object playback sound (sound from the sound source OBJ) and the simulated playback sound (sound simulated by the loudspeaker) to which the respective reverb components (initial reflection sound and reverberation sound) are added. Thus, the listener can also consider the reverb components and judge the accuracy of the reproduction of the object playback sound based on the simulated playback sound.

[0092] [Third Embodiment]

[0093] A sound field assist device and a sound field assist method according to a third embodiment of the present application will be described with reference to the drawings. Figure 11 is a functional block diagram showing the structure of a sound field assist system including a sound field assist device according to the third embodiment of the present application.

[0094] As Figure 11 The sound field assist device 10B according to the third embodiment differs from the sound field assist device 10 according to the first embodiment in that the posture detection section 70 is added. The other structure of the sound field assist device 10B is the same as that of the sound field assist device 10, and the description of the same parts is omitted.

[0095] The posture detection section 70 is attached to the listener's head and detects the posture of the listener's head. For example, the posture detection section 70 is a posture detection sensor of orthogonal three axes and is attached to the earphone 80. The posture detection section 70 outputs the detected posture of the listener's head to the binauralization processing section 50.

[0096] The binauralization processing section 50 performs binauralization processing on the object playback sound signal and the simulated playback sound signal using the result of the attitude detection of the head of the listener, i.e., the orientation of the face of the listener.

[0097] Thus, the sound field assist device 10B can perceive the object playback sound and the simulated playback sound corresponding to the orientation of the face of the listener. Therefore, the listener can perceive the object playback sound and the simulated playback sound corresponding to the orientation of the face of the listener in the object space while changing the orientation of the face. Thus, the listener can perceive the difference between the object playback sound and the simulated playback sound directly in multiple orientations in the object space, and can more accurately determine whether the simulated playback sound can accurately reproduce (simulate) the object playback sound and whether there is a sense of discomfort between the object playback sound and the simulated playback sound. In addition, as a result, the listener can more accurately reproduce the object playback sound through the simulated playback sound.

[0098] (Sound field assist method of the third embodiment)

[0099] Figure 12 is a flowchart showing a sound field assist method involved in the third embodiment of the present application. Figure 12 The sound field assist method shown in Figure 4 The sound field assist method shown in is supplemented with a flow of processing associated with the attitude detection of the head. In addition, the description of the same processing as Figure 12 The sound field assist method shown in is supplemented with a flow of processing associated with the attitude detection of the head. In addition, the description of the same processing as Figure 4 The sound field assist method shown in is supplemented with a flow of processing associated with the attitude detection of the head. In addition, the description of the same processing as

[0100] Figure 12 The sound field assist method shown in is supplemented with a flow of processing associated with the attitude detection of the head. In addition, the description of the same processing as Figure 4 The sound field assist method shown in is supplemented with a flow of processing associated with the attitude detection of the head. In addition, the description of the same processing as

[0101] The attitude detection section 70 detects the attitude of the head of the listener (S41).

[0102] The selection section 40 selects the object playback sound signal and the simulated playback sound signal by operation from the listener or the like (S15).

[0103] If the object playback sound signal is selected (S150: YES), the binauralization processing section 50 performs binauralization processing on the object playback sound signal using the detected attitude of the head (S461). If the simulated playback sound signal is selected (S150: NO), the binauralization processing section 50 performs binauralization processing on the simulated playback sound signal using the detected attitude of the head (S462).

[0104] The sound field assist device 10B performs the processing of step S17 using the sound signal on which the binauralization processing is performed.

[0105] As described above, the sound field assist method of the third embodiment is able to output the object playback sound and the simulated playback sound corresponding to the orientation of the face of the listener. Therefore, the listener is able to listen to the object playback sound and the simulated playback sound corresponding to the orientation of the face of the listener in contrast while changing the orientation of the face within the object space. Therefore, the listener is able to directly perceive the difference between the object playback sound and the simulated playback sound in a plurality of orientations within the object space, and is able to more accurately determine whether the simulated playback sound accurately reproduces (simulates) the object playback sound, and whether there is a sense of discomfort between the object playback sound and the simulated playback sound. In addition, as a result, the listener is able to more accurately reproduce the object playback sound through the simulated playback sound.

[0106] Further, the structure and the process of each of the above-described embodiments can be appropriately combined, and effects corresponding to each combination can be achieved.

[0107] Further, the description of the present embodiment is illustrative in all aspects and is not restrictive. The scope of the present application is not represented by the above-described embodiments but by the claims. Also, the scope of the present application includes all modifications within the equivalent meaning and range of the claims.

[0108] Explanation of Reference Signs

[0109] 10, 10A, 10B: sound field assist apparatus

[0110] 21: listener point setting section

[0111] 22: sound source position setting section

[0112] 23: speaker position setting section

[0113] 29: adjustment operation section

[0114] 30: simulated playback sound signal generation section

[0115] 40: selection section

[0116] 50: binauralization processing section

[0117] 60: reverberation processing section

[0118] 70: attitude detection section

[0119] 80: earphone

[0120] 90: object space

[0121] 99: virtual space

[0122] 100, 100A: GUI

[0123] 111: position relationship confirmation window

[0124] 111A: display window setting

[0125] 112: waveform confirmation window

[0126] 113, 116: operation member

[0127] 115: output state display window

[0128] 900: audiovisual point

Claims

1. A sound field assisting method, wherein, either position information of a sound source set on a virtual space or localization information of the sound source when a sound of the sound source is simulated by an output sound from a speaker set on an object space is selected, if the position information is selected, an object playback sound signal is generated based on the position information, if the localization information is selected, a simulation playback sound signal is generated based on the localization information, a sound image localization of the sound source realized by the speaker is adjusted using a sound based on the object playback sound signal and the simulation playback sound signal.

2. The sound field assisting method according to claim 1, wherein, the sound based on the object playback sound signal and the sound based on the simulation playback sound signal are compared, and the sound image localization is adjusted based on a comparison result.

3. The sound field assisting method according to claim 1 or 2, wherein, either an initial reflection sound or a reverberation sound is added to the sound based on the object playback sound signal and the sound based on the simulation playback sound signal.

4. The sound field assisting method according to claim 1 or 2, wherein, a listening position is set at the object space, binaural processing is set based on the position information or the localization information and the listening position, the sound after the binaural processing is output.

5. The sound field assisting method according to claim 4, wherein, an orientation of a face of a listener at the listening position is set, the binaural processing is set based on the position information or the localization information, the listening position, and the orientation of the face.

6. A sound field assisting apparatus, comprising: a selection section that selects either position information of a sound source set on a virtual space or localization information of the sound source when a sound of the sound source is simulated by an output sound from a speaker set on an object space, and outputs an object playback sound signal if the position information is selected; a simulation playback sound signal generation section that generates a simulation playback sound signal based on the localization information if the localization information is selected; and an adjustment section that adjusts a sound image localization of the sound source realized by the speaker using a sound based on the object playback sound signal and the simulation playback sound signal.

7. The sound field assisting apparatus according to claim 6, wherein, the adjustment section compares the sound based on the object playback sound signal and the sound based on the simulation playback sound signal, and adjusts the sound image localization based on a comparison result.

8. The sound field assisting apparatus according to claim 6 or 7, wherein, a reverb processing section that adds either an initial reflection sound or a reverberation sound to the sound based on the object playback sound signal and the sound based on the simulation playback sound signal is provided.

9. The sound field assisting apparatus according to claim 6 or 7, wherein, a listening point setting section that sets a listening position at the object space is provided. ​ a binauralization processing section that performs binauralization processing on a sound based on the object play sound signal or a sound based on the simulated play sound signal, based on the position information or the localization information and the audiovisual position, and outputs the binauralization-processed sound.

10. The sound field assist apparatus according to claim 9, wherein a posture detection section that detects an orientation of a face of an audiovisual person at the audiovisual position, the binauralization processing section sets the binauralization processing based on the position information or the localization information, the audiovisual position, and the orientation of the face.

11. A storage medium that is a computer-readable storage medium, storing a program that causes a computer to function as: a selection section that selects either position information of a sound source set on a virtual space or localization information of the sound source when a sound of the sound source is simulated by an output sound from a speaker set on an object space, generates an object play sound signal based on the position information and outputs the object play sound signal if the position information is selected, a simulated play sound signal generation section that generates a simulated play sound signal based on the localization information if the localization information is selected, and an adjustment section that adjusts a sound image localization of the sound source by the speaker using a sound based on the object play sound signal and the simulated play sound signal.

Images