Acoustic signal processing method, information generation method, computer program, and acoustic signal processing device

Abstract

A sound signal processing method includes: a first acquisition step of acquiring object information including first position information indicating a position of an object in a virtual space, first sound data indicating a first sound caused by the object, and first identification information indicating a processing means for the first sound data; the second position information indicates the position of a listener of the first sound in the virtual space; a first calculation step for calculating the distance between the object and the listener on the basis of first position information included in the acquired object information and the acquired second position information; a determination step for determining, on the basis of first identification information included in the acquired target information, a processing means for processing the first sound data from among a first processing means for processing the volume corresponding to the calculated distance and a second processing means for processing the volume corresponding to the calculated distance; a second processing means for processing the volume corresponding to the calculated distance in a form different from that of the first processing means; a first processing step of processing the first voice data by the determined processing means; and a first output step of outputting the processed first sound data.

Description

Technical Field

[0001] The present disclosure relates to an audio signal processing method and the like. Background Art

[0002] Patent Document 1 discloses a technique related to a stereophonic calculation method as a sound signal processing method. In this sound signal processing method, the volume (sound pressure) is controlled so as to change inversely proportional to the distance between a sound source and a listener (observer).

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2013-201577

[0006] Patent Document 2: International Publication No. 2021 / 180938

[0007] Non-patent literature

[0008] Non-patent document 1: Akiyoshi Iida, Physical 4-simulated sound waves and long-distance fields of aerodynamic noise (Physical 4-simulated sound waves and long-distance fields of aerodynamic noise) [online], [retrieved on June 21, 2020], Internet (URL: https: / / fluid.mech.kogakuin.ac.jp / ~iida / Lectures / master / aeroacoustic.pdf)

[0009] Non-patent document 2: The Society of Air Conditioning and Sanitary Engineers, Practical Knowledge of Air Conditioning Equipment Planning and Design (Revised 4th Edition), Ohmsha Publishing, Japan, March 24, 2017, p236

[0010] Non-patent document 3: Yoshinori Dobashi et al., Real-time rendering of aerodynamic sound using sound textures based on computational fluid dynamics, ACM Transactions on Graphics, Vol. 22, No. 3, p732-740 Summary of the invention

[0011] Problems to be solved by the invention

[0012] However, in the technology disclosed in Patent Document 1, it is sometimes difficult to provide a sense of presence to the listener.

[0013] Therefore, an object of the present disclosure is to provide an audio signal processing method and the like that can provide a sense of presence to a listener.

[0014] Means used to solve problems

[0015] A sound signal processing method according to a technical solution of the present disclosure includes: a first acquisition step of acquiring object information and second position information, wherein the object information includes first position information indicating the position of an object in a virtual space, first sound data indicating a first sound caused by the object, and first identification information indicating a means for processing the first sound data, and the second position information indicates the position of a listener of the first sound in the virtual space; a first calculation step of calculating the position between the object and the sound based on the first position information included in the acquired object information and the acquired second position information. the distance between the above-mentioned listeners; a determination step, based on the above-mentioned first identification information included in the above-mentioned object information obtained, determining a processing means for processing the above-mentioned first sound data from a first processing means and a second processing means, the above-mentioned first processing means performs volume processing corresponding to the above-mentioned calculated distance, and the above-mentioned second processing means performs volume processing corresponding to the above-mentioned calculated distance in a form different from the above-mentioned first processing means; a first processing step, processing the above-mentioned first sound data by the determined processing means; and a first output step, outputting the processed first sound data.

[0016] In addition, an information generation method related to a technical solution of the present invention includes: a second acquisition step, acquiring first sound data representing a first sound generated at a position associated with a listener's position in a virtual space, and first position information representing a position of an object in the above-mentioned virtual space; and a first generation step, generating first object audio information based on the above-mentioned first sound data obtained and the above-mentioned first position information obtained, the first object audio information including information related to the above-mentioned object and the above-mentioned first position information that causes the above-mentioned first sound to be reproduced at a position associated with the above-mentioned listener's position due to the above-mentioned object.

[0017] In addition, a sound signal processing method related to a technical solution of the present disclosure includes: a third acquisition step, obtaining the above-mentioned first object audio information generated by the above-mentioned information generation method, the above-mentioned first sound data obtained, and the second position information representing the above-mentioned position of the above-mentioned listener of the above-mentioned first sound; a second calculation step, calculating the distance between the above-mentioned object and the above-mentioned listener based on the above-mentioned first position information contained in the above-mentioned first object audio information obtained and the above-mentioned second position information obtained; a second processing step, processing the above-mentioned first sound data in a manner that the volume of the above-mentioned first sound attenuates as the calculated distance increases; and a second output step, outputting the processed above-mentioned first sound data.

[0018] In addition, an audio signal processing method related to a technical solution of the present disclosure includes: a third acquisition step, acquiring the above-mentioned first object audio information generated by the above-mentioned information generation method, the above-mentioned first sound data obtained, and the second position information representing the above-mentioned position of the above-mentioned listener of the above-mentioned first sound; a second calculation step, calculating the distance between the above-mentioned object radiating the above-mentioned wind and the above-mentioned listener based on the above-mentioned first position information contained in the above-mentioned first object audio information obtained and the above-mentioned second position information obtained; a second processing step, processing the above-mentioned first sound data in a manner that the volume of the above-mentioned first sound attenuates as the calculated distance increases; and a second output step, outputting the processed above-mentioned first sound data.

[0019] In addition, an audio signal processing method related to a technical solution of the present disclosure includes: a third acquisition step of acquiring the first object audio information generated by the information generation method, the first sound data obtained, and the second position information representing the position of the listener of the first sound; a second calculation step of calculating the distance between the object radiating the wind and the listener based on the first position information contained in the first object audio information obtained and the second position information obtained; a second processing step of processing the first sound data in a manner such that the volume of the first sound is attenuated according to the calculated distance and the unit distance when the calculated distance is larger than the unit distance represented by the unit distance information contained in the first object audio information obtained; and a second output step of outputting the processed first sound data.

[0020] In addition, a sound signal processing method related to a technical solution of the present disclosure includes: a third acquisition step of acquiring the first object audio information generated by the information generation method, the first sound data obtained, and the second position information indicating the position of the listener of the first sound; a second calculation step of calculating the distance between the object radiating the wind and the listener, and the direction between two points connecting the object and the listener based on the first position information included in the first object audio information obtained and the second position information obtained; a second processing step of processing the first sound data in a manner of controlling the volume of the first sound based on the angle between the front direction and the calculated direction between the two points and the characteristics indicated by the directivity information, and in a manner of attenuating the volume of the first sound according to the calculated distance and the unit distance when the calculated distance is larger than the unit distance indicated by the unit distance information included in the first object audio information obtained; and a second output step of outputting the processed first sound data.

[0021] In addition, a sound signal processing method related to a technical solution of the present disclosure includes: a third acquisition step, acquiring the above-mentioned first object audio information generated by the above-mentioned information generation method, the above-mentioned first sound data obtained, and the second object audio information that establishes an association between the above-mentioned first position information and the second sound data representing the second sound caused by the above-mentioned object; a second processing step, not convolving the first sound signal based on the above-mentioned first sound data obtained with the head-related transfer function processing that depends on the direction of arrival of the sound, and convolving the second sound signal based on the above-mentioned second sound data represented by the above-mentioned second object audio information obtained with the head-related transfer function processing that depends on the direction of arrival of the sound; and a second output step, outputting the unprocessed above-mentioned first sound signal and the processed above-mentioned second sound signal.

[0022] In addition, an audio signal processing method related to a technical solution of the present disclosure includes: a third acquisition step, acquiring the above-mentioned first object audio information generated by the above-mentioned information generation method, the above-mentioned first sound data obtained, and the second object audio information that establishes an association between the above-mentioned first position information and the second sound data representing the second sound caused by the above-mentioned object; a second processing step, processing the first sound signal based on the above-mentioned first sound data obtained in dependence on the direction of wind arrival, and convolving the second sound signal based on the above-mentioned second sound data represented by the above-mentioned second object audio information obtained with a head-related transfer function processing that depends on the direction of sound arrival; and a second output step, outputting the processed above-mentioned first sound signal and the processed above-mentioned second sound signal.

[0023] In addition, a sound signal processing method related to a technical solution of the present disclosure includes: a third acquisition step, obtaining the above-mentioned first object audio information generated by the above-mentioned information generation method, the above-mentioned first sound data obtained, and the third object audio information, the above-mentioned third object audio information establishes an association between the third position information representing the position of other objects in the above-mentioned virtual space and the third sound data representing the third sound generated at the above-mentioned position of the above-mentioned other objects, and the above-mentioned other objects are objects different from the above-mentioned objects; a second processing step, processing the first sound signal based on the above-mentioned first sound data obtained in dependence on the direction of wind arrival, and convolving the third sound signal based on the above-mentioned third sound data represented by the above-mentioned third object audio information obtained with a head-related transfer function processing depending on the direction of sound arrival; and a second output step, outputting the processed above-mentioned first sound signal and the processed above-mentioned third sound signal.

[0024] In addition, an information generation method related to a technical solution of the present disclosure includes: a fourth acquisition step of acquiring a generation position of a first wind blowing in a virtual space, a first wind direction of the first wind, and a first assumed wind speed as the wind speed of the first wind; a second generation step of generating fourth object audio information that establishes an association between the acquired generation position, the first wind direction, and the first assumed wind speed; a storage step of storing aerodynamic sound core information including representative wind speed and aerodynamic sound data, the aerodynamic sound data representing aerodynamic sound generated by the wind representing the wind speed reaching the ears of a listener in the virtual space; and a third output step of outputting the generated fourth object audio information and the stored aerodynamic sound core information.

[0025] In addition, the sound signal processing method of a technical solution related to the present disclosure includes: a fifth acquisition step, obtaining the above-mentioned fourth object audio information and the above-mentioned pneumatic sound core information output by the above-mentioned information generation method, and the second position information indicating the position of the above-mentioned listener in the above-mentioned virtual space; a third calculation step, calculating the distance between the above-mentioned generation position and the above-mentioned listener based on the above-mentioned generation position contained in the above-mentioned fourth object audio information obtained and the above-mentioned second position information obtained; a third processing step, processing the above-mentioned pneumatic sound data in a manner that the volume of the above-mentioned pneumatic sound attenuates as the calculated distance increases; and a fourth output step, outputting the above-mentioned pneumatic sound data after processing

[0026] In addition, an acoustic signal processing method related to a technical solution of the present disclosure includes: a fifth acquisition step of acquiring the fourth object audio information output by the information generation method and the pneumatic sound core information including data representing the distribution of the frequency components of the pneumatic sound, and the second position information representing the position of the listener in the virtual space; a third calculation step of calculating the distance between the generation position and the listener based on the generation position included in the fourth object audio information obtained and the second position information obtained, a third processing step of processing the pneumatic sound data in a manner such that the distribution of the frequency components of the pneumatic sound is shifted toward the low frequency side as the calculated distance increases; and a fourth output step of outputting the processed pneumatic sound data.

[0027] In addition, an information generation method related to a technical solution of the present disclosure includes: a sixth acquisition step of acquiring a second wind direction of a second wind blowing in a virtual space and a second assumed wind speed as the wind speed of the second wind; a third generation step of generating a fifth object audio information that associates the acquired second wind direction with the second assumed wind speed; a storage step of storing aerodynamic sound core information including representative wind speed and aerodynamic sound data, the aerodynamic sound data representing aerodynamic sound generated by the wind representing the wind speed reaching the ears of a listener in the virtual space; and a fifth output step of outputting the generated fifth object audio information and the stored aerodynamic sound core information.

[0028] In addition, an information generation method related to a technical solution of the present disclosure includes: a seventh acquisition step of acquiring a generation position of a first wind blowing in a virtual space, a first wind direction of the first wind, a first assumed wind speed as the wind speed of the first wind, a second wind direction of a second wind blowing in the virtual space, and a second assumed wind speed as the wind speed of the second wind; a fourth generation step of generating fourth object audio information that associates the acquired generation position, the first wind direction and the first assumed wind speed, and generating fifth object audio information that associates the acquired second wind direction with the second assumed wind speed; and a sixth output step of outputting the generated fourth object audio information and the generated fifth object audio information.

[0029] In addition, an audio signal processing method related to a technical solution of the present disclosure includes: an 8th acquisition step, obtaining the second position information indicating the position of the above-mentioned listener in the above-mentioned virtual space, and the above-mentioned 4th object audio information or the above-mentioned 5th object audio information output by the above-mentioned information generation method; a 4th processing step, when the above-mentioned 4th object audio information is obtained, the above-mentioned aerodynamic sound data contained in the above-mentioned aerodynamic sound core information is processed based on the above-mentioned position indicated by the above-mentioned 2nd position information obtained, and when the above-mentioned 5th object audio information is obtained, the above-mentioned aerodynamic sound data contained in the above-mentioned aerodynamic sound core information is not processed based on the above-mentioned position indicated by the above-mentioned 2nd position information obtained; and a 7th output step, outputting the above-mentioned aerodynamic sound data after processing.

[0030] Furthermore, a program according to one technical aspect of the present disclosure causes a computer to execute the above-mentioned sound signal processing method.

[0031] Furthermore, a program according to one technical solution of the present disclosure causes a computer to execute the above-mentioned information generating method.

[0032] In addition, a sound signal processing device according to a technical solution of the present disclosure comprises: a first acquisition unit for acquiring object information and second position information, wherein the object information includes first position information indicating the position of an object in a virtual space, first sound data indicating a first sound caused by the object, and first identification information indicating a means for processing the first sound data, and the second position information indicates the position of a listener of the first sound in the virtual space; and a first calculation unit for calculating the position of the first sound based on the first position information included in the acquired object information and the acquired second position information. the distance between the object and the listener; a determining unit, based on the first identification information included in the acquired object information, determining a processing means for processing the first sound data from a first processing means and a second processing means, the first processing means performing volume processing corresponding to the calculated distance, and the second processing means performing volume processing corresponding to the calculated distance in a form different from that of the first processing means; a first processing unit, processing the first sound data by the determined processing means; and a first output unit, outputting the processed first sound data.

[0033] In addition, these inclusive or specific technical solutions can also be implemented by systems, devices, methods, integrated circuits, computer programs, or non-temporary recording media such as computer-readable CD-ROMs, or by any combination of systems, devices, methods, integrated circuits, computer programs, and recording media.

[0034] Effects of the Invention

[0035] According to an audio signal processing method according to a technical solution of the present disclosure, a sense of presence can be brought to the listener. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a diagram for explaining a first example of aerodynamic sound (wind sound) generated along with the movement (position change) of an object.

[0037] Figure 2 This is a diagram for explaining a second example of aerodynamic sound (wind sound) generated along with the movement (position change) of an object.

[0038] Figure 3 This is a diagram for explaining the third example of aerodynamic sound (wind sound) generated along with the movement (position change) of an object.

[0039] Figure 4A This is a diagram for explaining aerodynamic sound generated when wind radiated from an object reaches the ears of a listener.

[0040] Figure 4B This is a diagram showing a method of measuring the attenuation of the volume of the second aerodynamic sound due to the distance from the wind source.

[0041] Figure 4C It means that Figure 4B The graph of the measurement results obtained by the experiment described in is a graph showing the frequency characteristics of the collected sounds.

[0042] Figure 4D This is a graph in which the frequency characteristics of the second pneumatic sound and the motor sound at each distance are converted into volume and plotted for each distance.

[0043] Figure 5A is a diagram illustrating a listener's personal space.

[0044] Figure 5B This is a diagram showing an immersive audio reproduction system as an example of a system to which the audio processing or decoding processing of the present disclosure can be applied.

[0045] Figure 5C This is a functional block diagram showing the configuration of an encoding device as an example of the encoding device of the present disclosure.

[0046] Figure 5D This is a functional block diagram showing the configuration of a decoding device as an example of the decoding device of the present disclosure.

[0047] Figure 5E This is a functional block diagram showing the configuration of an encoding device as another example of the encoding device of the present disclosure.

[0048] Fig. 5FThis is a functional block diagram showing the configuration of a decoding device as another example of the decoding device of the present disclosure.

[0049] Figure 5G It means as Figure 5D or Fig. 5F A functional block diagram of the structure of a decoder of an example of a decoder in FIG.

[0050] Figure 5H It means as Figure 5D or Fig. 5F A functional block diagram of the structure of another example of a decoder in the decoder.

[0051] Fig.5I This is a diagram showing an example of the physical structure of the sound signal processing device.

[0052] Figure 5J This is a diagram showing an example of the physical structure of an encoding device.

[0053] Figure 6 This is a block diagram showing the functional structure of the sound signal processing device according to the embodiment.

[0054] Figure 7 This is a flowchart of operation example 1 of the sound signal processing device according to the embodiment.

[0055] Figure 8 1 is a diagram showing a bat and a listener as objects related to Action Example 1.

[0056] Fig. 9 This is a block diagram showing the functional structure of the sound signal processing device related to Modification Example 1.

[0057] Fig.10 This is a diagram showing four other persons and a listener related to Modification Example 1.

[0058] Fig.11 This is a flowchart of Operation Example 2 of the sound signal processing device according to Modification Example 1.

[0059] Fig.12 This is a block diagram showing the functional structure of the sound signal processing device related to Modification Example 2.

[0060] Fig.13 This is a diagram showing an object and multiple sounds related to Modification Example 2.

[0061] Fig.14 This is a flowchart of Operation Example 3 of the sound signal processing device according to Modification Example 2.

[0062] Fig.15 This is a diagram showing a case where the target according to Modification Example 2 is an electric fan.

[0063] Fig.16 This is a diagram showing a case where the object according to Modification Example 2 is a zombie.

[0064] Fig.17 This is a block diagram showing the functional structure of the information generating device and the sound signal processing device related to the third modification.

[0065] Fig.18 This is a diagram showing an electric fan and a listener as objects related to Modification Example 3.

[0066] Fig.19 It is a diagram used to explain the directivity information and unit distance information related to the modified example 3.

[0067] Fig. 20 This is a diagram for explaining the processing of the second processing unit related to Modification Example 3.

[0068] Fig.21 This is a diagram used to explain other processing of the second processing unit related to Modification Example 3.

[0069] Fig. 22 This is a flowchart of operation example 4 of the information generating device related to modification example 3.

[0070] Fig.23 This is a flowchart of Operation Example 5 of the sound signal processing device according to Modification Example 3.

[0071] Fig.24 This is a block diagram showing the functional structure of the information generating device and the sound signal processing device related to Modification Example 4.

[0072] Fig.25 This is a diagram for explaining the processing on the first audio data related to Modification Example 4.

[0073] Fig.26 This is a flowchart of Operation Example 6 of the sound signal processing device according to Modification Example 4.

[0074] Fig. 27 This is a flowchart of Operation Example 7 of the sound signal processing device according to Modification Example 4.

[0075] Fig.28 This is a block diagram showing the functional structure of the information generating device and the sound signal processing device related to Modification Example 5.

[0076] Fig.29 This is a flowchart of Operation Example 8 of the sound signal processing device according to Modification Example 5.

[0077] Fig.30 This is a block diagram showing the functional structure of the information generating device and the sound signal processing device related to the sixth modification.

[0078] Fig.31 This is a flowchart of operation example 9 of the information generating device related to modification example 6.

[0079] Fig.32 This is a flowchart of operation example 10 of the sound signal processing device according to modification example 6.

[0080] Fig.33 This is a flowchart of Operation Example 11 of the sound signal processing device according to Modification Example 6.

[0081] Fig.34 This is a block diagram showing the functional structure of the information generating device and the sound signal processing device related to the seventh modification.

[0082] Fig.35 This is a diagram showing an example of an image displayed on the display unit according to Modification Example 7.

[0083] Fig.36 This is a flowchart of operation example 12 of the information generating device related to modification example 7.

[0084] Fig.37 This is a flowchart of operation example 13 of the sound signal processing device related to modification example 7.

[0085] Fig.38 This is a block diagram showing the functional structure of the information generating device and the sound signal processing device related to Modification Example 8.

[0086] Fig.39 This is a flowchart of operation example 14 of the information generating device related to modification example 8.

[0087] Fig.40 It is used to illustrate Figure 5G and Figure 5H A functional block diagram of a case where a rendering unit performs pipeline processing and a diagram showing an example of steps. DETAILED DESCRIPTION

[0088] (Understanding that is the basis of the present disclosure)

[0089] Conventionally, there is known an acoustic signal processing method for controlling the volume (sound pressure) of sound heard by a listener in a virtual space.

[0090] Patent document 1 discloses a technique related to a stereophonic sound calculation method as a sound signal processing method. In this sound signal processing method, the volume (sound pressure) is controlled so as to change inversely proportional to the distance between the sound source and the listener (observer). More specifically, the volume is controlled so as to attenuate inversely proportional to the increase in the distance. As a result, the listener can recognize the distance between the object emitting the sound, that is, the sound source, and the listener himself.

[0091] Such controlled sound is used in applications such as virtual reality (VR (Virtual Reality)) and augmented reality (AR (Augmented Reality)) for reproducing three-dimensional sound in a space where a user (listener) exists.

[0092] Furthermore, in real space, there are known examples of sounds in which the volume of a sound heard by a listener attenuates in accordance with conditions different from the inverse proportion of the distance between the object emitting the sound and the listener himself.

[0093] As such sounds, the following two sounds are exemplified.

[0094] The first sound is aerodynamic sound (so-called wind noise) generated as the object moves (see Non-Patent Document 1). Aerodynamic sound (wind noise) is a sound based on pressure changes such as vortex trains of air generated by wind hitting the object or the object moving in the air.

[0095] Figure 1 1 is a diagram for explaining the first example of aerodynamic sound (wind noise) generated by the movement (position change) of an object. Figure 1 In the video, the object is a baseball bat B. When the object (bat B) moves (changes in position), that is, when bat B is swung, a wind sound is generated. By hearing the wind sound, the listener L can recognize that bat B is swung. In the real space, the volume of the wind sound decays as the distance between bat B and listener L increases, more specifically, decays at the square of the distance between bat B and listener L.

[0096] If the technology disclosed in Patent Document 1 is applied to such wind noise in a virtual space, the listener L can hear the wind noise whose volume is controlled to attenuate inversely proportional to the distance between the bat B and the listener L. That is, the wind noise to which the technology disclosed in Patent Document 1 is applied in a virtual space will be a sound different from the wind noise heard by the listener L in a real space. In a virtual space, if the listener L hears the wind noise to which the technology disclosed in Patent Document 1 is applied, since the wind noise is a sound different from the wind noise heard by the listener L in a real space, the listener L feels awkward and it is difficult for the listener L to get a sense of presence. Therefore, an acoustic signal processing method or the like that can give the listener L a sense of presence is required.

[0097] Figure 2 2 is a diagram for explaining a second example of aerodynamic sound (wind noise) generated by the movement (position change) of an object. Figure 2In the video, the object is ambulance A. When the object (ambulance A) moves (changes in position), that is, when ambulance A is driving, wind noise is generated. By hearing the wind noise, listener L can recognize that ambulance A is driving. In the real space, like the wind noise caused by the bat B, the volume of the wind noise decays as the distance between ambulance A and listener L increases. More specifically, it decays at the square of the distance between ambulance A and listener L.

[0098] Furthermore, in Figure 2 In the example, the ambulance A is also an object that emits a siren. The volume of the siren in the real space attenuates inversely proportional to the increase in the distance between the ambulance A and the listener L.

[0099] A case where the technology disclosed in Patent Document 1 is applied to both the wind noise and the whistle sound caused by an ambulance A in a virtual space is studied.

[0100] In this case, regarding the whistle sound, the listener L hears the whistle sound whose volume is controlled to attenuate in inverse proportion to the distance between the ambulance A and the listener L. That is, the whistle sound in the virtual space using the technology disclosed in Patent Document 1 becomes the same sound as the whistle sound heard by the listener L in the real space, and the listener L is less likely to feel uncomfortable.

[0101] On the other hand, in this case, regarding the sound of the wind, the listener L also hears the sound of the wind whose volume is controlled to attenuate inversely proportional to the distance between the ambulance A and the listener L. That is, the sound of the wind in the virtual space using the technology disclosed in Patent Document 1 becomes a sound different from the sound of the wind heard by the listener L in the real space. If the listener L hears the sound of the wind using the technology disclosed in Patent Document 1 in the virtual space, the listener L feels awkward and it is difficult for the listener L to get a sense of presence. Therefore, in such a case, an acoustic signal processing method that can give the listener L a sense of presence is also required.

[0102] In addition, as described above, the ambulance A is an object that generates a plurality of sounds (the whistle sound and the sound of the wind).

[0103] Figure 3 This is a diagram for explaining the third example of aerodynamic sound (wind noise) generated by the movement (position change) of an object. Figure 3 , an electric fan F is exemplified as such an object.

[0104] When the multiple blades of the electric fan F move (rotate), the wind whistling sound is generated. Like the wind whistling sound caused by the bat B and the wind whistling sound caused by the ambulance A, the volume of the wind whistling sound caused by the electric fan F decays as the distance between the electric fan F and the listener L increases. More specifically, it decays at the square of the distance between the electric fan F and the listener L. In addition, the electric fan F can also be an object that emits the sound of the motor of the electric fan F, that is, the motor sound. The volume of the motor sound decays inversely proportional to the increase in the distance between the electric fan F and the listener L in the real space.

[0105] As described above, the electric fan F is also an object that generates multiple sounds (motor sound and wind noise).

[0106] Therefore, similarly to the case where the object is the ambulance A, when the technology disclosed in Patent Document 1 is applied to both the wind noise and the motor noise caused by the electric fan F in the virtual space, the listener L feels uncomfortable and it is difficult for the listener L to get a sense of presence. Figure 3 As shown, when the listener L moves from position (a) to position (b), that is, when the distance between the electric fan F and the listener L changes, the sense of discomfort increases. Therefore, an acoustic signal processing method or the like that can give the listener L a sense of presence is required.

[0107] As described above, the wind noise is described as the first sound. Next, the aerodynamic sound generated by the wind radiated from the object reaching the ear of the listener L is described as the second sound (refer to non-patent documents 2 and 3). In addition, the ear refers to at least one of the auricle and the outer ear.

[0108] In the prior art including the technology disclosed in Patent Document 1, object audio information is used. Object audio information is data that associates, for example, a sound signal (sound data) representing a sound with position information representing the position where the sound is generated. Figure 4A Example of representation.

[0109] Figure 4A This is a diagram for explaining aerodynamic sound generated by wind W emitted from an object reaching the ear of listener L. This aerodynamic sound as the second sound is a sound generated, for example, according to the shape of the ear of listener L when wind W brought by electric fan F as the object reaches listener L. In addition, for the purpose of identification below, aerodynamic sound (wind rustling sound) generated along with the movement (position change) of the object is sometimes recorded as the first aerodynamic sound, and aerodynamic sound generated by wind W emitted from the object and reaching the ear of listener L hitting the ear of listener L as the second sound is recorded as the second aerodynamic sound.

[0110] exist Figure 4AIn the example, the sound caused by the electric fan F as the object is included in Figure 3 Here, we focus on the motor sound and the second pneumatic sound.

[0111] For example, if the target audio information is used for motor sound, the sound signal (sound data) representing the motor sound is associated with the position information representing the position where the motor sound is generated. In the target audio information, the position where the motor sound is generated is the position of the electric fan F.

[0112] Furthermore, if the target audio information is used for the second aerodynamic sound, the sound signal (sound data) representing the second aerodynamic sound is associated with the position information representing the position where the second aerodynamic sound is generated. In the target audio information, the position where the second aerodynamic sound is generated is the position of the listener L.

[0113] Here, regarding the motor sound, by controlling the volume so as to attenuate in inverse proportion to the distance between the electric fan F, where the motor sound is generated, and the listener L, the listener L can hear the motor sound with little discomfort.

[0114] In addition, regarding the second aerodynamic sound, there is no definite conclusion as for the motor sound or the first aerodynamic sound about how the volume attenuates depending on the distance between the electric fan F, which is the location where the wind W is generated, and the listener L. However, from the real experience of life, it is obvious that the volume increases and decreases depending on the distance. Therefore, the inventors of this application experimentally clarified the trend of its increase and decrease using an electric fan and a pseudo microphone.

[0115] Figure 4B This is a diagram showing how the volume of the second aerodynamic sound is attenuated by the distance from the wind source (i.e., the location where the wind W is generated (electric fan F)). Figure 4B As shown, pseudo microphones (hereinafter referred to as microphones) are arranged at various positions 1m, 2m, 4m, etc. from the electric fan to collect the sound generated when the wind W blows. Since the net in front of the electric fan F is removed, the collected sound is only the second pneumatic sound and the motor sound, and the first pneumatic sound brought by the net is excluded.

[0116] Figure 4C It means in Figure 4B The graph of the measurement results obtained by the experiment described in is a graph showing the frequency characteristics of the collected sound. More specifically, Figure 4C (a) shows the microphone is placed at a distance of 1m. Figure 4C (b) shows the case where the microphone is placed at a distance of 2m. Figure 4C(c) shows the case where the microphone is placed at a position of 4m. The line (1st line) in which the frequency components below 1kHz are greatly raised represents the frequency characteristics of the sound collected in the direction where the wind W blows on the microphone. Therefore, the 1st line represents the combined frequency characteristics of the motor sound and the second aerodynamic sound. The other line (2nd line) represents the frequency characteristics of the sound collected in the direction where the wind W does not blow on the microphone. Therefore, the 2nd line represents the frequency characteristics of the motor sound only. In all the graphs, the 1st line and the 2nd line overlap in the frequency components above about 1kHz, so the frequency components of the second aerodynamic sound can be represented by the components below 1kHz of the 1st line. Of course, the frequency components of the motor sound can be represented by the 2nd line as a whole.

[0117] Figure 4D The frequency characteristics of the second air sound and motor sound determined in this way at each distance are converted into volume and plotted at each distance. Generally speaking, the motor sound tends to attenuate in proportion to the distance (inversely proportional to the distance). On the other hand, the second air sound tends to attenuate at the 2.5th power of the distance (inversely proportional to the 2.5th power of the distance). Figure 4D In the Figure 4B (The observation data at the positions of 50 cm and 3 m are not shown in the figure). Therefore, regarding the second aerodynamic sound, by controlling so that the volume is attenuated by the 2.5th power of the distance between the electric fan F, which is the position where the wind W is generated, and the listener L, the listener L can hear the second aerodynamic sound that is not easy to feel awkward. Alternatively, it is also possible to establish an association between the distance and the frequency characteristics of the second aerodynamic sound at the distance, obtain the frequency characteristics with the distance as an index, and control the volume. Of course, since it is known that the wind speed attenuates according to the distance, it is also possible to establish an association between the wind speed and the frequency characteristics of the second aerodynamic sound at the wind speed, obtain the frequency characteristics with the wind speed calculated according to the distance as an index, and control the volume.

[0118] In addition, the inventors of the present application expected that the exponent of the distance attenuation of the second aerodynamic sound is 4 (inversely proportional to the fourth power of the distance). This is considered to be because the second aerodynamic sound originates from the aerodynamic sound, the so-called cavity sound, generated when the wind blows on an object with an uneven surface, and it is said that the volume of the cavity sound is amplified in proportion to the fourth power of the wind speed, and the wind speed is inversely proportional to the distance, so it can be expected to be inversely proportional to the fourth power of the distance. However, it can be seen from the above-mentioned experiments that the exponent of the distance attenuation of the second aerodynamic sound is about 2.5. This is considered to be because the auricle is not a simple cavity, but a cavity with a parabolic shape, so it can capture the wind W more effectively. On the other hand, since the shape of the auricle varies from person to person, the above exponent cannot be limited to 2.5. However, since the second aerodynamic sound originates from the cavity sound, it can be expected that the above exponent does not exceed 4.

[0119] On the other hand, regarding the second aerodynamic sound, the position where the second aerodynamic sound is generated is the position of the ear of the listener L. Therefore, even if the listener L moves, the distance between the position where the second aerodynamic sound is generated, that is, the position of the ear of the listener L, and the position of the listener L is always constant, so the volume of the second aerodynamic sound cannot be controlled to attenuate according to the distance. Therefore, if the object audio information related to the conventional technology is used, the listener L will hear the second aerodynamic sound that feels awkward. Therefore, an acoustic signal processing method that can give the listener L a sense of presence is required.

[0120] Furthermore, it is known that there is a personal space in the real space. Figure 5A This is a diagram for explaining the personal space of the listener L.

[0121] Personal space is the limit of how close others are allowed to get to oneself (here, listener L), that is, a psychological defense line. This means that for listener L in the virtual space, there is a sense of distance between listener L and other people that cannot be expressed by physical distance alone. Personal space is classified into four categories: close distance (less than 45 cm), individual distance (greater than 45 cm and less than 120 cm), social distance (greater than 120 cm and less than 360 cm), and public distance (greater than 360 cm).

[0122] In order to present the sense of distance between the listener L and other persons that cannot be expressed by the physical distance alone in a virtual space, a case where the technology disclosed in Patent Document 1 is uniformly applied to the sounds emitted by all other persons is studied.

[0123] In this case, the listener L hears a sound whose volume is controlled to attenuate inversely proportional to the distance between the other person and the listener L, regardless of the relationship between the other person and the listener L. That is, regardless of whether the other person is a person with a high degree of intimacy or a person with a low degree of intimacy with respect to the listener L, the sound of the other person is uniformly controlled to attenuate.

[0124] Therefore, it is difficult to present the personal space corresponding to the listener L in the virtual space and the sense of distance between the listener L and other people that cannot be expressed only by the physical distance, so the listener L will feel awkward and it is difficult for the listener L to get a sense of presence. Therefore, an audio signal processing method that can give the listener L a sense of presence is required.

[0125] Therefore, the sound signal processing method of the first technical solution of the present disclosure includes: a first acquisition step of acquiring object information and second position information, wherein the object information includes first position information indicating the position of an object in a virtual space, first sound data indicating a first sound caused by the object, and first identification information indicating a means for processing the first sound data, and the second position information indicates the position of a listener of the first sound in the virtual space; a first calculation step of calculating the position of the first sound based on the first position information included in the acquired object information and the acquired second position information. the distance between the object and the listener; a determination step, based on the first identification information included in the acquired object information, determining a processing means for processing the first sound data from a first processing means and a second processing means, the first processing means performing volume processing corresponding to the calculated distance, and the second processing means performing volume processing corresponding to the calculated distance in a form different from that of the first processing means; a first processing step, processing the first sound data by the determined processing means; and a first output step, outputting the processed first sound data.

[0126] Thus, the processing means for changing the volume of the first sound according to the first identification information can make the first sound in the virtual space listened by the listener similar to the first sound in the real space listened by the listener, and more specifically, reproduce the first sound in the real space. Therefore, the listener is less likely to feel awkward and can get a sense of presence. That is, the sound signal processing method can give the listener a sense of presence.

[0127] In addition, for example, the sound signal processing method of the second technical solution of the present disclosure is the sound signal processing method of the first technical solution, the first processing means is a processing means for processing the first sound data in a manner such that the volume attenuates inversely proportionally with an increase in the calculated distance, and the second processing means is a processing means for processing the first sound data in a manner such that the volume increases or decreases in a manner different from that of the first processing means as the calculated distance increases.

[0128] Thus, since the first processing means for processing the first sound data in a manner that causes the volume to attenuate inversely as the distance increases or the second processing means for increasing or decreasing the volume in a manner different from the first processing means as the distance increases is used according to the first identification information, the first sound in the virtual space listened to by the listener becomes a sound more similar to the first sound in the real space listened to by the listener. Therefore, the listener is less likely to feel awkward, and the listener can get a better sense of presence. That is, the sound signal processing method can give the listener a better sense of presence.

[0129] In addition, for example, the sound signal processing method related to the third technical solution of the present disclosure is the sound signal processing method related to the first or second technical solution, and the above-mentioned object information obtained includes: second sound data, representing a second sound caused by the above-mentioned object, and the second sound is a sound different from the above-mentioned first sound; and second identification information, representing a processing means for the above-mentioned second sound data. In the above-mentioned determination step, based on the above-mentioned second identification information included in the above-mentioned object information obtained, a processing means for processing the above-mentioned second sound data is determined between the above-mentioned first processing means and the above-mentioned second processing means. In the above-mentioned first processing step, the above-mentioned second sound data is processed by the determined above-mentioned processing means. In the above-mentioned first output step, the processed second sound data is output, and the above-mentioned object is an object associated with multiple sound data including the above-mentioned first sound data and the above-mentioned second sound data.

[0130] Thus, the processing means for changing the volume of the second sound can be used according to the second identification information, so that the second sound in the virtual space listened to by the listener also becomes a sound similar to the second sound in the real space listened to by the listener. More specifically, according to the calculated distance, the volume balance of the first sound and the second sound changes like the volume balance that occurs in the real space. Therefore, the listener is less likely to feel awkward, and the listener can get a better sense of presence. That is, the sound signal processing method can give the listener a better sense of presence.

[0131] In addition, for example, the sound signal processing method of the fourth technical solution of the present disclosure is the sound signal processing method of the second technical solution, and the second processing means is a processing means for processing the first sound data in a manner that causes the volume to decay by the xth power of the distance, where x≠1.

[0132] Thus, in the first processing step, the second processing means for processing the first audio data so that the volume is attenuated by the xth power of the distance can be used.

[0133] Furthermore, for example, the sound signal processing method according to the fifth technical solution of the present disclosure is the sound signal processing method according to the fourth technical solution, and the first identification information indicates that the processing means for the first sound data is the second processing means, and the value of x.

[0134] Thus, the first identification information can indicate that the processing means is the second processing means, and in the first processing step, the first audio data can be processed according to the value of x indicated by the first identification information.

[0135] In addition, for example, the sound signal processing method of the sixth technical solution of the present disclosure is the sound signal processing method of the fourth technical solution. When the first sound is a pneumatic sound generated as the object moves, the first identification information indicates that the processing means for the first sound data is the second processing means, and x is α, α is a real number, and α satisfies the following formula: α>1.

[0136] Thus, in the first processing step, when the first sound is a pneumatic sound (first pneumatic sound), the first sound data can be processed according to α which is the value of x indicated by the first identification information.

[0137] In addition, for example, the sound signal processing method of the 7th technical solution of the present disclosure is the sound signal processing method of the 6th technical solution. When the above-mentioned first sound is aerodynamic sound generated by the wind radiated from the above-mentioned object reaching the ears of the above-mentioned listener, the above-mentioned first identification information indicates that the processing means for the above-mentioned first sound data is the above-mentioned second processing means, and the above-mentioned x is β, the above-mentioned β is a real number, and the above-mentioned β satisfies the following formula: β>2.

[0138] Thus, in the first processing step, when the first sound is aerodynamic sound (second aerodynamic sound) generated by wind radiated from the object reaching the listener's ears, the first sound data can be processed according to β which is the value of x indicated by the first identification information.

[0139] In addition, for example, the sound signal processing method according to the eighth technical solution of the present disclosure is the sound signal processing method according to the seventh technical solution, and the above α and the above β satisfy the following formula: α<β.

[0140] Thus, in the first processing step, the first audio data can be processed using α or β satisfying α<β.

[0141] Furthermore, for example, the sound signal processing method according to the ninth technical aspect of the present disclosure is the sound signal processing method according to the seventh or eighth technical aspect, and includes a first receiving step of receiving an operation of designating a value of the above-mentioned α or the above-mentioned β from a user.

[0142] Thus, in the first processing step, the first audio data can be processed using the value of α or β specified by the user.

[0143] In addition, for example, the sound signal processing method related to the 10th technical solution of the present disclosure is the sound signal processing method related to the 2nd technical solution, and the above-mentioned first identification information is information indicating whether to execute the above-mentioned first processing means; in the above-mentioned decision step, whether to execute the above-mentioned first processing means is determined based on the above-mentioned first identification information obtained, and regardless of whether the above-mentioned first processing means is executed, it is determined to execute the above-mentioned second processing means; the above-mentioned second processing means is a processing means for processing the above-mentioned first sound data in a manner that the volume becomes a prescribed value when the calculated above-mentioned distance is within a prescribed threshold value.

[0144] Thus, in the first processing step, the volume becomes a specified value only when the distance is within a specified threshold, thereby presenting an unrealistic effect, and the second processing means can process the first sound data so as to also give a natural distance attenuation effect that occurs in reality.

[0145] Furthermore, for example, the sound signal processing method according to the eleventh technical aspect of the present disclosure is the sound signal processing method according to the tenth technical aspect, wherein the predetermined threshold value is a value corresponding to the personal space.

[0146] Thus, in the first processing step, the first audio data can be processed using the predetermined threshold value that is a value corresponding to the personal space, so that a psychological sense of distance that cannot be expressed only by the distance attenuation effect based on the physical distance can be presented.

[0147] Furthermore, for example, the sound signal processing method according to the twelfth technical solution of the present disclosure is the sound signal processing method according to the tenth or eleventh technical solution, and includes a first acceptance step of accepting an operation from a user specifying that the predetermined threshold is a first designated value.

[0148] Thus, in the first processing step, the first audio data can be processed using the first designated value designated by the user.

[0149] In addition, for example, the information generation method related to the 13th technical solution of the present disclosure includes: a second acquisition step, acquiring first sound data representing a first sound generated at a position associated with a listener's position in a virtual space, and first position information representing the position of an object in the above-mentioned virtual space; and a first generation step, generating first object audio information based on the above-mentioned first sound data obtained and the above-mentioned first position information obtained, the first object audio information including information related to the above-mentioned object and the above-mentioned first position information that causes the above-mentioned first sound to be reproduced at a position associated with the above-mentioned listener's position due to the above-mentioned object.

[0150] Thus, it is possible to generate first object audio information that associates first sound data representing a first sound generated at a position associated with a listener's position due to an object with the position of the object. When the first object audio information is used in an acoustic signal processing method, the first sound data is processed in such a way that the volume of the first sound attenuates as the distance between the object and the listener increases, so that the first sound in the virtual space listened to by the listener becomes a sound similar to the first sound in the real space listened to by the listener, and more specifically, becomes a sound that reproduces the first sound in the real space. Therefore, the listener is less likely to feel awkward, and the listener can get a sense of presence. That is, the information generation method can give the listener a sense of presence.

[0151] In addition, for example, the information generation method related to the 14th technical solution of the present disclosure is the information generation method related to the 13th technical solution, the above-mentioned object radiates wind; the above-mentioned listener bathes in the above-mentioned radiated wind; the above-mentioned first sound is aerodynamic sound generated by the above-mentioned wind radiated from the above-mentioned object reaching the ears of the above-mentioned listener.

[0152] This realizes an information generation method that can make the first sound become aerodynamic sound (second aerodynamic sound) generated by wind radiated from the object and reaching the ears of the listener.

[0153] In addition, for example, the information generation method related to the 15th technical solution of the present disclosure is the information generation method related to the 14th technical solution, and in the above-mentioned first generation step, the above-mentioned first object audio information also including unit distance information is generated; the above-mentioned unit distance information is information including a distance serving as a reference, i.e., a unit distance, and pneumatic sound data representing the above-mentioned pneumatic sound at a position away from the above-mentioned position of the above-mentioned object by the above-mentioned unit distance.

[0154] Thus, the first object audio information including the unit distance information can be generated. When the first object audio information is used in the sound signal processing method, the first sound (second aerodynamic sound) in the virtual space listened to by the listener becomes a sound more similar to the first sound (second aerodynamic sound) in the real space listened to by the listener based on the unit distance and aerodynamic sound data. Therefore, the listener is less likely to feel awkward, and the listener can get a better sense of presence. That is, the information generation method can bring a better sense of presence to the listener.

[0155] In addition, for example, the information generation method related to the 16th technical solution of the present disclosure is the information generation method related to the 15th technical solution, and in the above-mentioned first generation step, the above-mentioned first object audio information also including directionality information is generated; the above-mentioned directionality information is information representing characteristics corresponding to the direction of the above-mentioned wind being radiated; the above-mentioned aerodynamic sound data represented by the above-mentioned unit distance information is data representing the above-mentioned aerodynamic sound at a position away from the above-mentioned position of the above-mentioned object by the above-mentioned unit distance in the front direction of the above-mentioned wind radiated by the above-mentioned object represented by the above-mentioned directionality information.

[0156] Thus, the first object audio information including the directional information can be generated. When the first object audio information is used in the sound signal processing method, the first sound (second aerodynamic sound) in the virtual space listened to by the listener becomes a sound more similar to the first sound (second aerodynamic sound) in the real space listened to by the listener based on the unit distance, aerodynamic sound data and directional information. Therefore, the listener is less likely to feel awkward, and the listener can get a better sense of presence. That is, the information generation method can bring a better sense of presence to the listener.

[0157] In addition, for example, the information generation method related to the 17th technical solution of the present disclosure is an information generation method related to any one of the 13th to 16th technical solutions, and in the above-mentioned first generation step, the above-mentioned first object audio information is generated which also includes flag information, and the above-mentioned flag information indicates whether to perform convolution processing of the first sound signal based on the above-mentioned first sound data depending on the head-related transfer function processing of the direction of arrival of the sound when the above-mentioned first sound is reproduced, and the above-mentioned first sound data represents the above-mentioned first sound generated from the above-mentioned object.

[0158] Thus, the first object audio information including the marker information can be generated. When the first object audio information is used in the sound signal processing method, sometimes the first sound signal based on the first sound data is convolved with the head-related transfer function, so that the first sound in the virtual space listened by the listener becomes a sound more similar to the first sound in the real space listened by the listener. Therefore, the listener is less likely to feel awkward, and the listener can get a better sense of presence. That is, the information generation method can give the listener a better sense of presence.

[0159] In addition, for example, the sound signal processing method related to the 18th technical solution of the present disclosure includes: a third acquisition step, obtaining the above-mentioned first object audio information generated by the information generation method related to the 13th technical solution, the above-mentioned first sound data obtained, and the second position information representing the above-mentioned position of the above-mentioned listener of the above-mentioned first sound; a second calculation step, calculating the distance between the above-mentioned object and the above-mentioned listener based on the above-mentioned first position information contained in the above-mentioned first object audio information obtained and the above-mentioned second position information obtained; a second processing step, processing the above-mentioned first sound data in a manner that the volume of the above-mentioned first sound is attenuated as the calculated distance increases; and a second output step, outputting the processed above-mentioned first sound data.

[0160] Thus, in the third acquisition step, the first object audio information that associates the first sound data representing the first sound generated by the object at a position associated with the position of the listener with the position of the object can be obtained. Therefore, by processing the first sound data in a manner such that the volume of the first sound decreases as the distance between the object and the listener increases, the first sound in the virtual space listened to by the listener becomes a sound similar to the first sound in the real space listened to by the listener, and more specifically, becomes a sound that reproduces the first sound in the real space. Therefore, the listener is less likely to feel awkward, and the listener can get a sense of presence. That is, the sound signal processing method can give the listener a sense of presence.

[0161] In addition, for example, the sound signal processing method related to the 19th technical solution of the present disclosure includes: a third acquisition step, obtaining the above-mentioned first object audio information generated by the information generation method related to the 14th technical solution, the above-mentioned first sound data obtained, and the second position information representing the above-mentioned position of the above-mentioned listener of the above-mentioned first sound; a second calculation step, calculating the distance between the above-mentioned object radiating the above-mentioned wind and the above-mentioned listener based on the above-mentioned first position information contained in the above-mentioned first object audio information obtained and the above-mentioned second position information obtained; a second processing step, processing the above-mentioned first sound data in a manner that the volume of the above-mentioned first sound is attenuated as the calculated distance increases; and a second output step, outputting the processed above-mentioned first sound data.

[0162] This realizes an acoustic signal processing method that can convert the first sound into aerodynamic sound (second aerodynamic sound) generated by wind radiated from the object and reaching the ears of the listener.

[0163] In addition, for example, the sound signal processing method related to the 20th technical solution of the present disclosure includes: a third acquisition step, obtaining the above-mentioned first object audio information generated by the information generation method related to the 15th technical solution, the above-mentioned first sound data obtained, and the second position information representing the above-mentioned position of the above-mentioned listener of the above-mentioned first sound; a second calculation step, calculating the distance between the above-mentioned object radiating the above-mentioned wind and the above-mentioned listener based on the above-mentioned first position information contained in the above-mentioned first object audio information obtained and the above-mentioned second position information obtained; a second processing step, processing the above-mentioned first sound data in a manner such that when the calculated distance is larger than the above-mentioned unit distance represented by the above-mentioned unit distance information contained in the above-mentioned first object audio information obtained, the volume of the above-mentioned first sound is attenuated according to the calculated distance and the above-mentioned unit distance; and a second output step, outputting the processed above-mentioned first sound data.

[0164] Thus, in the third acquisition step, the first object audio information including the unit distance information can be acquired. Therefore, the first sound (second aerodynamic sound) in the virtual space listened to by the listener becomes a sound more similar to the first sound (second aerodynamic sound) in the real space listened to by the listener based on the unit distance and aerodynamic sound data. Therefore, the listener is less likely to feel awkward, and the listener can get a better sense of presence. That is, the sound signal processing method can give the listener a better sense of presence.

[0165] In addition, for example, the sound signal processing method of the 21st technical solution of the present disclosure includes: a third acquisition step of acquiring the first object audio information generated by the information generation method of the 16th technical solution, the first sound data obtained, and the second position information indicating the position of the listener of the first sound, a second calculation step of calculating the distance between the object radiating the wind and the listener based on the first position information contained in the first object audio information obtained and the second position information obtained, and calculating the distance between the object and the listener. the direction between two points connected by the front direction; a second processing step of controlling the volume of the first sound based on the angle between the front direction and the calculated direction between the two points and the characteristic represented by the directivity information, and processing the first sound data in a manner such that when the calculated distance is larger than the unit distance represented by the unit distance information included in the obtained first object audio information, the volume of the first sound is attenuated according to the calculated distance and the unit distance; and a second output step of outputting the processed first sound data.

[0166] Thus, in the third acquisition step, the first object audio information including the directional information can be acquired. Therefore, the first sound (second aerodynamic sound) in the virtual space listened to by the listener becomes a sound more similar to the first sound (second aerodynamic sound) in the real space listened to by the listener based on the unit distance, aerodynamic sound data and directional information. Therefore, the listener is less likely to feel awkward, and the listener can get a better sense of presence. That is, the sound signal processing method can bring a better sense of presence to the listener.

[0167] In addition, for example, the sound signal processing method related to the 22nd technical scheme of the present disclosure includes: a third acquisition step, obtaining the above-mentioned first object audio information generated by the information generation method related to any one of the technical schemes 13 to 16, the above-mentioned first sound data obtained, and the second object audio information that establishes an association between the above-mentioned first position information and the second sound data representing the second sound caused by the above-mentioned object; a second processing step, not convolving the first sound signal based on the above-mentioned first sound data obtained with the head-related transfer function processing that depends on the direction of arrival of the sound, and convolving the second sound signal based on the above-mentioned second sound data represented by the above-mentioned second object audio information obtained with the head-related transfer function processing that depends on the direction of arrival of the sound; and a second output step, outputting the unprocessed above-mentioned first sound signal and the processed above-mentioned second sound signal.

[0168] Thus, since the second sound signal based on the second sound data is convolved with the head-correlated transfer function, the second sound in the virtual space listened to by the listener becomes a sound similar to the second sound in the real space listened to by the listener, more specifically, a sound that reproduces the second sound in the real space. Therefore, the listener is less likely to feel awkward, and the listener can get a better sense of presence. That is, the sound signal processing method can give the listener a better sense of presence.

[0169] In addition, for example, the sound signal processing method related to the 23rd technical solution of the present disclosure includes: a third acquisition step, obtaining the above-mentioned first object audio information generated by the information generation method related to any one of the technical solutions 14 to 16, the above-mentioned first sound data obtained, and the second object audio information that establishes an association between the above-mentioned first position information and the second sound data representing the second sound caused by the above-mentioned object; a second processing step, processing the first sound signal based on the above-mentioned first sound data obtained in dependence on the direction of wind arrival, and convolving the second sound signal based on the above-mentioned second sound data represented by the above-mentioned second object audio information obtained with a head-related transfer function processing that depends on the direction of sound arrival; and a second output step, outputting the processed above-mentioned first sound signal and the processed above-mentioned second sound signal.

[0170] Thus, since the first sound signal based on the first sound data is processed depending on the direction of the wind, the first sound (second aerodynamic sound) in the virtual space listened to by the listener becomes a sound similar to the first sound (second aerodynamic sound) in the real space listened to by the listener, more specifically, it becomes a sound that reproduces the first sound (second aerodynamic sound) in the real space. Furthermore, since the convolution head correlation transfer function is performed on the second sound signal based on the second sound data, the second sound in the virtual space listened to by the listener becomes a sound similar to the second sound in the real space listened to by the listener, more specifically, it becomes a sound that reproduces the second sound in the real space. Therefore, the listener is less likely to feel awkward, and the listener can get a better sense of presence. That is, the sound signal processing method can bring a better sense of presence to the listener.

[0171] In addition, for example, the sound signal processing method related to the 24th technical scheme of the present disclosure includes: a third acquisition step, obtaining the above-mentioned first object audio information generated by the information generation method of any one of the technical schemes related to the 14th to 16th technical schemes, the above-mentioned first sound data obtained, and the third object audio information, the above-mentioned third object audio information establishes an association between the third position information representing the position of other objects in the above-mentioned virtual space and the third sound data representing the third sound generated at the above-mentioned position of the above-mentioned other objects, and the above-mentioned other objects are objects different from the above-mentioned objects; a second processing step, processing the first sound signal based on the above-mentioned first sound data obtained in dependence on the direction of wind arrival, and convolving the third sound signal based on the above-mentioned third sound data represented by the above-mentioned third object audio information obtained with a head-related transfer function processing dependent on the direction of sound arrival; and a second output step, outputting the processed above-mentioned first sound signal and the processed above-mentioned third sound signal.

[0172] Thus, when multiple objects including the object and other objects are set in the virtual space, the first sound (second aerodynamic sound) and the third sound in the virtual space listened to by the listener become similar to the first sound (second aerodynamic sound) and the third sound in the real space listened to by the listener. Therefore, the listener is less likely to feel awkward, and the listener can get a better sense of presence. That is, the sound signal processing method can give the listener a better sense of presence.

[0173] In addition, for example, the information generation method related to the 25th technical solution of the present disclosure includes: a 4th acquisition step, acquiring the generation position of the first wind blowing in the virtual space, the first wind direction of the above-mentioned first wind, and the first assumed wind speed as the wind speed of the above-mentioned first wind; a 2nd generation step, generating the 4th object audio information that establishes an association between the above-mentioned generation position, the above-mentioned first wind direction and the above-mentioned first assumed wind speed; a storage step, storing aerodynamic sound core information including representative wind speed and aerodynamic sound data, the above-mentioned aerodynamic sound data representing the aerodynamic sound generated by the wind representing the above-mentioned wind speed reaching the ears of the listener in the above-mentioned virtual space; and a 3rd output step, outputting the generated above-mentioned 4th object audio information and the stored above-mentioned aerodynamic sound core information.

[0174] Thus, it is possible to generate the fourth object audio information that establishes an association between the generation position, the first wind direction, and the first assumed wind speed. When the fourth object audio information is used in the sound signal processing method, the aerodynamic sound data is processed in a manner such that the volume of the aerodynamic sound (the second aerodynamic sound) attenuates as the distance between the object and the listener increases. The aerodynamic sound (the second aerodynamic sound) in the virtual space listened to by the listener becomes a sound similar to the aerodynamic sound (the second aerodynamic sound) in the real space listened to by the listener, and more specifically, becomes a sound that reproduces the aerodynamic sound (the second aerodynamic sound) in the real space. Therefore, the listener is less likely to feel awkward, and the listener can get a sense of presence. That is, the information generation method can bring a sense of presence to the listener.

[0175] In addition, for example, the information generation method related to the 26th technical solution of the present disclosure is the information generation method related to the 25th technical solution, and the above-mentioned first assumed wind speed is the wind speed of the above-mentioned first wind at a position that is a unit distance away from the above-mentioned generation position as a reference distance in the direction of the above-mentioned first wind direction.

[0176] Thus, the wind speed of the first wind at a position away from the unit distance can be used as the first assumed wind speed.

[0177] Furthermore, for example, the information generation method according to the 27th technical solution of the present disclosure is the information generation method according to the 26th technical solution, and includes a second acceptance step of accepting an operation from a user to designate that the unit distance is a second designated value.

[0178] Thereby, the fourth target audio information can be generated using the unit distance which is the second designated value designated by the user.

[0179] In addition, for example, the information generation method related to the 28th technical solution of the present disclosure is an information generation method related to the 26th or 27th technical solution, including a second acceptance step of accepting an operation of specifying directional information from a user, the above-mentioned directional information representing characteristics corresponding to the direction of the above-mentioned first wind, and in the above-mentioned second generation step, the above-mentioned fourth object audio information is generated by associating the above-mentioned generation position, the above-mentioned first wind direction and the above-mentioned first assumed wind speed with the above-mentioned directional information represented by the accepted operation.

[0180] Thereby, the fourth target audio information in which the generation position, the first wind direction, the first assumed wind speed, and the directional information designated by the user are associated can be generated.

[0181] In addition, for example, the sound signal processing method related to the 29th technical solution of the present disclosure includes: a third acquisition step, obtaining the above-mentioned fourth object audio information and the above-mentioned aerodynamic sound core information output by the information generation method of any one of the technical solutions related to the 26th to 28th technical solutions, and the second position information indicating the position of the above-mentioned listener in the above-mentioned virtual space; a third calculation step, calculating the distance between the above-mentioned generation position and the above-mentioned listener based on the above-mentioned generation position contained in the obtained above-mentioned fourth object audio information and the obtained above-mentioned second position information; a third processing step, processing the above-mentioned aerodynamic sound data in a manner that the volume of the above-mentioned aerodynamic sound attenuates as the calculated distance increases; and a fourth output step, outputting the above-mentioned aerodynamic sound data after processing.

[0182] Thus, in the fifth acquisition step, the fourth object audio information that establishes an association between the generation position, the first wind direction, and the first assumed wind speed can be obtained. Therefore, by processing the aerodynamic sound data in a manner such that the volume of the aerodynamic sound (the second aerodynamic sound) attenuates as the distance between the object and the listener increases, the aerodynamic sound (the second aerodynamic sound) in the virtual space listened to by the listener becomes a sound similar to the aerodynamic sound (the second aerodynamic sound) in the real space listened to by the listener, and more specifically, becomes a sound that reproduces the aerodynamic sound (the second aerodynamic sound) in the real space. Therefore, the listener is less likely to feel awkward, and the listener can get a sense of presence. That is, the sound signal processing method can bring a sense of presence to the listener.

[0183] In addition, for example, the sound signal processing method related to the 30th technical solution of the present disclosure is the sound signal processing method related to the 29th technical solution. In the above-mentioned third processing step, the above-mentioned aerodynamic sound data is processed based on the wind speed of the above-mentioned first wind when it reaches the ears of the above-mentioned listener, that is, the wind speed reaching the ears; regarding the above-mentioned wind speed reaching the ears, the more the calculated distance increases, the slower the above-mentioned wind speed.

[0184] Thus, the aerodynamic sound data is processed based on the wind speed reaching the ear, so that the aerodynamic sound (second aerodynamic sound) in the virtual space listened to by the listener becomes a sound more similar to the aerodynamic sound (second aerodynamic sound) in the real space listened to by the listener. Therefore, the listener is less likely to feel awkward, and the listener can get a better sense of presence. That is, the sound signal processing method can give the listener a better sense of presence.

[0185] In addition, for example, the sound signal processing method of the 31st technical solution of the present disclosure is the sound signal processing method of the 30th technical solution, and the above-mentioned wind speed reaching the ear is a value attenuated by the zth power of the value obtained by dividing the calculated distance by the above-mentioned unit distance.

[0186] Thus, a more accurate wind speed reaching the ear is calculated, so that the aerodynamic sound (second aerodynamic sound) in the virtual space listened by the listener becomes a sound more similar to the aerodynamic sound (second aerodynamic sound) in the real space listened by the listener. Therefore, the listener is less likely to feel awkward and can get a better sense of presence. That is, the sound signal processing method can give the listener a better sense of presence.

[0187] In addition, for example, the sound signal processing method according to the 32nd technical solution of the present disclosure is the sound signal processing method according to the 31st technical solution, and the above z satisfies the following formula: z=1.

[0188] Thus, a more accurate wind speed reaching the ear is calculated, so that the aerodynamic sound (second aerodynamic sound) in the virtual space listened by the listener becomes a sound more similar to the aerodynamic sound (second aerodynamic sound) in the real space listened by the listener. Therefore, the listener is less likely to feel awkward and can get a better sense of presence. That is, the sound signal processing method can give the listener a better sense of presence.

[0189] In addition, for example, the sound signal processing method related to the 33rd technical solution of the present disclosure is the sound signal processing method related to the 31st technical solution, and in the above-mentioned third processing step, the above-mentioned aerodynamic sound data is processed in a manner such that the volume of the above-mentioned aerodynamic sound is attenuated by the γth power of the value obtained by dividing the above-mentioned representative wind speed by the above-mentioned wind speed reaching the ear.

[0190] Thus, the pneumatic sound data is processed so that the volume of the pneumatic sound (second pneumatic sound) becomes a more accurate volume, so the pneumatic sound (second pneumatic sound) in the virtual space listened by the listener becomes a sound more similar to the pneumatic sound (second pneumatic sound) in the real space listened by the listener. Therefore, the listener is less likely to feel awkward, and the listener can get a better sense of presence. That is, the sound signal processing method can give the listener a better sense of presence.

[0191] In addition, for example, the sound signal processing method according to the 34th technical solution of the present disclosure is the sound signal processing method according to the 33rd technical solution, and the above-mentioned γ and the above-mentioned z satisfy the following formula: γ×z<4.

[0192] Thus, a more accurate wind speed reaching the ear is calculated, so that the aerodynamic sound (second aerodynamic sound) in the virtual space listened by the listener becomes a sound more similar to the aerodynamic sound (second aerodynamic sound) in the real space listened by the listener. Therefore, the listener is less likely to feel awkward and can get a better sense of presence. That is, the sound signal processing method can give the listener a better sense of presence.

[0193] In addition, for example, the sound signal processing method related to the 35th technical scheme of the present disclosure includes: a 5th acquisition step, obtaining the above-mentioned 4th object audio information output by the information generation method of any one of the technical schemes related to the 26th to 28th technical schemes and the above-mentioned pneumatic sound core information including data representing the distribution of the frequency components of the above-mentioned pneumatic sound, and the second position information representing the position of the above-mentioned listener in the above-mentioned virtual space; a 3rd calculation step, calculating the distance between the above-mentioned generation position and the above-mentioned listener based on the above-mentioned generation position contained in the obtained 4th object audio information and the obtained 2nd position information, a 3rd processing step, processing the above-mentioned pneumatic sound data in a manner such that the distribution of the above-mentioned frequency components of the above-mentioned pneumatic sound is shifted toward the low frequency side as the calculated distance increases; and a 4th output step, outputting the processed above-mentioned pneumatic sound data.

[0194] Thus, in the fifth acquisition step, the fourth object audio information that establishes an association between the generation position, the first wind direction, and the first assumed wind speed can be obtained. Therefore, by processing the aerodynamic sound data in a manner such that the distribution of the frequency components of the aerodynamic sound (the second aerodynamic sound) shifts toward the low frequency side as the distance between the object and the listener increases, the aerodynamic sound (the second aerodynamic sound) in the virtual space listened to by the listener becomes a sound similar to the aerodynamic sound (the second aerodynamic sound) in the real space listened to by the listener, and more specifically, becomes a sound that reproduces the aerodynamic sound (the second aerodynamic sound) in the real space. Therefore, the listener is less likely to feel awkward, and the listener can get a sense of presence. That is, the sound signal processing method can bring a sense of presence to the listener.

[0195] In addition, for example, the sound signal processing method related to the 36th technical solution of the present disclosure is the sound signal processing method related to the 35th technical solution. In the above-mentioned third processing step, the above-mentioned aerodynamic sound data is processed based on the wind speed of the above-mentioned first wind when it reaches the ears of the above-mentioned listener, that is, the wind speed reaching the ears; regarding the above-mentioned wind speed reaching the ears, the more the calculated distance increases, the slower the above-mentioned wind speed.

[0196] Thus, the aerodynamic sound data is processed based on the wind speed reaching the ear, so that the aerodynamic sound (second aerodynamic sound) in the virtual space listened to by the listener becomes a sound more similar to the aerodynamic sound (second aerodynamic sound) in the real space listened to by the listener. Therefore, the listener is less likely to feel awkward, and the listener can get a better sense of presence. That is, the sound signal processing method can give the listener a better sense of presence.

[0197] In addition, for example, the sound signal processing method of the 37th technical solution of the present disclosure is the sound signal processing method of the 36th technical solution, and the above-mentioned wind speed reaching the ear is a value attenuated by the zth power of the value obtained by dividing the calculated distance by the unit distance.

[0198] Thus, a more accurate wind speed reaching the ear is calculated, so that the aerodynamic sound (second aerodynamic sound) in the virtual space listened by the listener becomes a sound more similar to the aerodynamic sound (second aerodynamic sound) in the real space listened by the listener. Therefore, the listener is less likely to feel awkward and can get a better sense of presence. That is, the sound signal processing method can give the listener a better sense of presence.

[0199] In addition, for example, the sound signal processing method according to the 38th technical solution of the present disclosure is the sound signal processing method according to the 37th technical solution, and the above z satisfies the following formula: z=1.

[0200] Thus, a more accurate wind speed reaching the ear is calculated, so that the aerodynamic sound (second aerodynamic sound) in the virtual space listened by the listener becomes a sound more similar to the aerodynamic sound (second aerodynamic sound) in the real space listened by the listener. Therefore, the listener is less likely to feel awkward and can get a better sense of presence. That is, the sound signal processing method can give the listener a better sense of presence.

[0201] In addition, for example, the sound signal processing method of the 39th technical solution of the present disclosure is the sound signal processing method of any one of the 36th to 38th technical solutions, and in the above-mentioned third processing step, the above-mentioned aerodynamic sound data is processed in a manner such that the distribution of the above-mentioned frequency components of the above-mentioned aerodynamic sound is shifted toward a frequency that is the reciprocal multiple of the value obtained by dividing the above-mentioned representative wind speed by the above-mentioned wind speed reaching the ear.

[0202] Thus, a more accurate wind speed reaching the ear is calculated, so that the aerodynamic sound (second aerodynamic sound) in the virtual space listened by the listener becomes a sound more similar to the aerodynamic sound (second aerodynamic sound) in the real space listened by the listener. Therefore, the listener is less likely to feel awkward and can get a better sense of presence. That is, the sound signal processing method can give the listener a better sense of presence.

[0203] In addition, for example, the information generation method of the 40th technical solution of the present disclosure includes: a 6th acquisition step, acquiring the second wind direction of the second wind blowing in the virtual space, and a second assumed wind speed as the wind speed of the above-mentioned second wind; a 3rd generation step, generating the 5th object audio information that establishes an association between the acquired above-mentioned second wind direction and the above-mentioned second assumed wind speed; a storage step, storing aerodynamic sound core information including representative wind speed and aerodynamic sound data, the above-mentioned aerodynamic sound data representing the aerodynamic sound generated by the wind representing the above-mentioned wind speed reaching the ears of the listener in the above-mentioned virtual space; and a 5th output step, outputting the generated above-mentioned 5th object audio information and the stored above-mentioned aerodynamic sound core information.

[0204] Thus, the fifth object audio information that associates the second wind direction with the second assumed wind speed can be generated. When the fifth object audio information is used in the sound signal processing method, it is possible to reproduce wind with an uncertain source of wind (natural wind blowing outdoors), and by processing the aerodynamic sound data not based on the position represented by the second position information, the aerodynamic sound (second aerodynamic sound) caused by the second wind in the virtual space listened to by the listener becomes a sound similar to the aerodynamic sound (second aerodynamic sound) caused by the second wind in the real space listened to by the listener, and more specifically, it becomes a sound that reproduces the aerodynamic sound (second aerodynamic sound) caused by the second wind in the real space. Therefore, the listener is less likely to feel awkward, and the listener can get a sense of presence. That is, the information generation method can give the listener a sense of presence.

[0205] In addition, for example, the information generation method related to the 41st technical solution of the present disclosure includes: a 7th acquisition step, acquiring the generation position of the first wind blowing in the virtual space, the first wind direction of the first wind, the first assumed wind speed as the wind speed of the first wind, the second wind direction of the second wind blowing in the virtual space, and the second assumed wind speed as the wind speed of the second wind; a 4th generation step, generating the 4th object audio information that establishes an association between the obtained generation position, the first wind direction and the first assumed wind speed, and generating the 5th object audio information that establishes an association between the obtained second wind direction and the second assumed wind speed; and a 6th output step, outputting the generated 4th object audio information and the generated 5th object audio information.

[0206] Thus, it is possible to generate the fourth object audio information that associates the generation position, the first wind direction, and the first assumed wind speed, and the fifth object audio information that associates the second wind direction and the second assumed wind speed, so that two types of wind can be generated in the same virtual space, namely, wind whose generation source can be determined (electric fan, exhaust hole, wind hole, etc.) and wind whose generation source cannot be determined (natural breeze, storm, etc.). Furthermore, when the fourth object audio information is used in the sound signal processing method, by processing the aerodynamic sound data based on the position represented by the second position information, the aerodynamic sound (second aerodynamic sound) caused by the first wind in the virtual space listened to by the listener becomes a sound similar to the aerodynamic sound (second aerodynamic sound) caused by the first wind in the real space listened to by the listener, and more specifically, becomes a sound that reproduces the aerodynamic sound (second aerodynamic sound) caused by the first wind in the real space. Furthermore, when the fifth object audio information is used in the sound signal processing method, by processing the aerodynamic sound data based on the position indicated by the second position information, the aerodynamic sound (second aerodynamic sound) caused by the second wind in the virtual space listened by the listener becomes a sound similar to the aerodynamic sound (second aerodynamic sound) caused by the second wind in the real space listened by the listener, and more specifically, becomes a sound that reproduces the aerodynamic sound (second aerodynamic sound) caused by the second wind in the real space. Therefore, the listener is less likely to feel awkward, and the listener can get a sense of presence. That is, the information generation method can give the listener a sense of presence.

[0207] In addition, for example, the information generation method related to the 42nd technical solution of the present disclosure is the information generation method related to the 41st technical solution. In the above-mentioned 6th output step, when the above-mentioned generation position of the above-mentioned 1st wind is in the above-mentioned virtual space, the generated above-mentioned 4th object audio information is output.

[0208] Therefore, the information generating method can determine whether the fourth object audio information can be output according to the generation position.

[0209] In addition, for example, the information generation method related to the 43rd technical solution of the present disclosure is the information generation method related to the 42nd technical solution. In the above-mentioned 6th output step, when the above-mentioned generation position of the above-mentioned 1st wind is not in the above-mentioned virtual space, the generated above-mentioned 5th object audio information is output.

[0210] Therefore, the information generating method can determine whether the fifth object audio information can be output according to the generation position.

[0211] In addition, for example, the information generation method related to the 44th technical solution of the present disclosure is an information generation method related to the 43rd technical solution, including a storage step, in which aerodynamic sound core information including data representing wind speed and aerodynamic sound is stored, and the above-mentioned aerodynamic sound data represents the aerodynamic sound generated by the wind representing the above-mentioned wind speed reaching the ears of the listener in the above-mentioned virtual space, and in the above-mentioned 6th output step, the stored aerodynamic sound core information is output.

[0212] Thus, when the pneumatic sound data included in the output pneumatic sound core information is used in the sound signal processing method, the pneumatic sound core information can be applied to the first wind and the second wind in common, so the memory area for storing the pneumatic sound core information can be reduced, and the pneumatic sound (second pneumatic sound) caused by the first wind in the virtual space listened to by the listener becomes a sound similar to the pneumatic sound (second pneumatic sound) caused by the first wind in the real space listened to by the listener, and the pneumatic sound (second pneumatic sound) caused by the second wind in the virtual space listened to by the listener becomes a sound similar to the pneumatic sound (second pneumatic sound) caused by the second wind in the real space listened to by the listener. Therefore, the listener is less likely to feel awkward, and the listener can get a sense of presence. That is, the information generation method can give the listener a sense of presence.

[0213] In addition, for example, the information generation method related to the 45th technical solution of the present disclosure is an information generation method related to the 44th technical solution, including: a display step of displaying an image that establishes an association between the wind speed and the words expressing the above wind speed; and a third acceptance step of accepting a first operation of specifying the above wind speed represented by the above image displayed as the above first assumed wind speed, and accepting a second operation of specifying the above wind speed represented by the above image displayed as the above second assumed wind speed.

[0214] Thereby, the wind speed designated by the user can be used as the first assumed wind speed, and the wind speed designated by the user can be used as the second assumed wind speed.

[0215] In addition, for example, the sound signal processing method related to the 43rd technical solution of the present disclosure includes: the 8th acquisition step, obtaining the second position information indicating the position of the above-mentioned listener in the above-mentioned virtual space, and the above-mentioned 4th object audio information or the above-mentioned 5th object audio information output by the information generation method related to the 44th technical solution; the 4th processing step, when the above-mentioned 4th object audio information is obtained, the above-mentioned aerodynamic sound data included in the above-mentioned aerodynamic sound core information is processed based on the above-mentioned position indicated by the above-mentioned second position information obtained, and when the above-mentioned 5th object audio information is obtained, the above-mentioned aerodynamic sound data included in the above-mentioned aerodynamic sound core information is not processed based on the above-mentioned position indicated by the above-mentioned second position information obtained; and the 7th output step, outputting the above-mentioned aerodynamic sound data after processing.

[0216] Thus, in the eighth acquisition step, the fourth object audio information or the fifth object audio information can be acquired. Therefore, by processing the pneumatic sound data based on the position indicated by the second position information, the pneumatic sound (second pneumatic sound) caused by the first wind in the virtual space listened by the listener becomes a sound similar to the pneumatic sound (second pneumatic sound) caused by the first wind in the real space listened by the listener, and more specifically, becomes a sound that reproduces the pneumatic sound (second pneumatic sound) caused by the first wind in the real space. Furthermore, by processing the pneumatic sound data not based on the position indicated by the second position information, the pneumatic sound (second pneumatic sound) caused by the second wind in the virtual space listened by the listener becomes a sound similar to the pneumatic sound (second pneumatic sound) caused by the second wind in the real space listened by the listener, and more specifically, becomes a sound that reproduces the pneumatic sound (second pneumatic sound) caused by the second wind in the real space. Therefore, the listener is less likely to feel awkward, and the listener can get a sense of presence. That is, the sound signal processing method can give the listener a sense of presence.

[0217] Furthermore, a computer program according to a 47th technical solution of the present disclosure is a computer program for causing a computer to execute the above-described sound signal processing method.

[0218] Thereby, the computer can execute the above-mentioned sound signal processing method according to the computer program.

[0219] Furthermore, the computer program according to the 48th technical solution of the present disclosure is a computer program for causing a computer to execute the information generating method described above.

[0220] Thereby, the computer can execute the above-mentioned information generating method according to the computer program.

[0221] In addition, the sound signal processing device of the 49th technical solution of the present disclosure comprises: a first acquisition unit, which acquires object information and second position information, the object information includes first position information indicating the position of an object in a virtual space, first sound data indicating a first sound caused by the object, and first identification information indicating a processing method for the first sound data, the second position information indicating the position of a listener of the first sound in the virtual space; and a first calculation unit, which calculates the first position information contained in the acquired object information and the acquired second position information. the distance between the object and the above-mentioned listener; a determining unit, which determines a processing means for processing the above-mentioned first sound data from a first processing means and a second processing means based on the above-mentioned first identification information included in the above-mentioned object information obtained, the above-mentioned first processing means processing the volume corresponding to the calculated above-mentioned distance, and the above-mentioned second processing means processing the volume corresponding to the above-mentioned distance calculated in a form different from the above-mentioned first processing means; a first processing unit, which processes the above-mentioned first sound data by the determined above-mentioned processing means; and a first output unit, which outputs the processed above-mentioned first sound data.

[0222] Thus, the processing means for changing the volume of the first sound can be used according to the first identification information, so that the first sound in the virtual space listened to by the listener becomes a sound similar to the first sound in the real space listened to by the listener, and more specifically, becomes a sound that reproduces the first sound in the real space. Therefore, the listener is less likely to feel awkward and the listener can get a sense of presence. That is, the sound signal processing device can give the listener a sense of presence.

[0223] Furthermore, these inclusive or specific technical solutions may also be implemented by systems, devices, methods, integrated circuits, computer programs, or non-temporary recording media such as computer-readable CD-ROMs, or by any combination of systems, devices, methods, integrated circuits, computer programs, and recording media.

[0224] Hereinafter, the embodiments will be described in detail with reference to the drawings.

[0225] In addition, the embodiments described below are all inclusive or specific examples. The numerical values, shapes, materials, components, configuration positions and connection forms of components, steps, and the order of steps shown in the following embodiments are examples and are not intended to limit the present disclosure.

[0226] In addition, in the following description, elements are sometimes given ordinals such as 1st and 2nd. These ordinals are given to elements for the purpose of identifying the elements and do not necessarily correspond to a meaningful order. These ordinals may be replaced as appropriate, may be newly given, or may be removed.

[0227] In addition, each figure is a schematic diagram and does not necessarily illustrate the exact figure. Therefore, the scales and the like in each figure are not necessarily the same. In each figure, the same reference numerals are given to substantially the same structure, and repeated descriptions are omitted or simplified.

[0228] In this specification, terms such as vertical and other terms indicating the relationship between elements and numerical ranges do not indicate only strict expressions, but mean substantially equivalent ranges, for example, expressions including differences of about several percentage points.

[0229] (Implementation Method)

[0230] [Examples of Devices to Which the Sound Processing Technology or Encoding / Decoding Technology of the Present Disclosure Can Be Applied]

[0231] (Stereo sound reproduction system)

[0232] Figure 5B 1 is a diagram showing an immersive audio reproduction system A0000 as an example of a system to which the audio processing or decoding processing of the present disclosure can be applied. The immersive audio reproduction system A0000 includes an audio signal processing device A0001 and an audio presentation device A0002.

[0233] The sound signal processing device A0001 performs sound processing on the sound signal emitted by the virtual sound source, and generates a sound signal after sound processing to prompt the listener (i.e., the receiver). The sound signal is not limited to the human voice, as long as it is an audible sound. For example, sound processing is a signal processing performed on the sound signal in order to reproduce one or more sound-related effects received from the time when the sound is emitted to the time when the listener hears it. The sound signal processing device A0001 performs sound processing based on information that records the causes of the above-mentioned sound-related effects. Spatial information includes, for example, information indicating the position of the sound source, the listener, and the surrounding objects, information indicating the shape of the space, parameters related to the propagation of sound, etc. The sound signal processing device A0001 is, for example, a PC (Personal Computer), a smart phone, a tablet computer, or a game console.

[0234] The sound-processed signal is prompted to the listener (user) from the sound prompting device A0002. The sound prompting device A0002 is connected to the sound signal processing device A0001 via wireless or wired communication. The sound signal generated by the sound signal processing device A0001 after sound processing is transmitted to the sound prompting device A0002 via wireless or wired communication. In the case where the sound prompting device A0002 is composed of multiple devices such as a device for the right ear and a device for the left ear, the multiple devices synchronously prompt the sound by communicating between the multiple devices or each of the multiple devices communicates with the sound signal processing device A0001. The sound prompting device A0002 is, for example, headphones, earplugs, a head-mounted display worn on the listener's head, or a surround speaker composed of multiple fixed speakers.

[0235] Furthermore, the stereo sound reproduction system A0000 may be used in combination with an image presentation device or a stereoscopic image presentation device that visually provides an ER (Extended Reality) experience including AR / VR.

[0236] in addition, Figure 5B The system configuration example shows that the sound signal processing device A0001 and the sound prompting device A0002 are different devices, but the stereo sound reproduction system to which the sound signal processing method or decoding method disclosed in the present invention can be applied is not limited to Figure 5B For example, the sound signal processing device A0001 may be included in the sound prompting device A0002, and the sound prompting device A0002 may perform both sound processing and sound prompting. In addition, the sound signal processing device A0001 and the sound prompting device A0002 may share the sound processing described in the present disclosure, or a server connected to the sound signal processing device A0001 or the sound prompting device A0002 via a network may implement part or all of the sound processing described in the present disclosure.

[0237] In addition, although the above description refers to the sound signal processing device A0001, when the sound signal processing device A0001 performs sound processing by decoding a bit stream generated by encoding data of at least a part of spatial information used in a sound signal or sound processing, the sound signal processing device A0001 may also be referred to as a decoding device.

[0238] (Example of encoding device)

[0239] Figure 5C It is a functional block diagram showing the structure of an encoding device A0100 which is an example of the encoding device of the present disclosure.

[0240] Input data A0101 is data to be encoded including spatial information and / or a sound signal input to encoder A0102. The details of spatial information will be described later.

[0241] The encoder A0102 encodes the input data A0101 to generate encoded data A0103. The encoded data A0103 is, for example, a bit stream generated by encoding processing.

[0242] The memory A0104 stores the encoded data A0103. The memory A0104 may be, for example, a hard disk or an SSD (Solid-State Drive), or may be other memory.

[0243] In addition, in the above description, a bit stream generated by encoding processing is cited as an example of the encoded data A0103 stored in the memory A0104, but it can also be data other than a bit stream. For example, the encoding device A0100 can also store the converted data generated by converting the bit stream into a specified data format in the memory A0104. The converted data can also be, for example, a file or a multiplexed stream storing one or more bit streams. Here, the file is a file having a file format such as ISOBMFF (ISO Base Media File Format). In addition, the encoded data A0103 can also be in the form of multiple packets generated by dividing the above-mentioned bit stream or file. In the case of converting the bit stream generated by the encoder A0102 into data different from the bit stream, the encoding device A0100 can also have a conversion unit not shown in the figure, and the conversion process can also be performed by a CPU (Central Processing Unit).

[0244] (Example of a decoding device)

[0245] Figure 5D It is a functional block diagram showing the structure of a decoding device A0110 which is an example of a decoding device according to the present disclosure.

[0246] The memory A0114 stores, for example, the same data as the coded data A0103 generated by the coding device A0100. The memory A0114 reads the stored data and inputs it as input data A0113 of the decoder A0112. The input data A0113 is, for example, a bit stream to be decoded. The memory A0114 may be, for example, a hard disk or an SSD, or other memory.

[0247] In addition, the decoding device A0110 may use the transformed data generated by transforming the read data instead of the data stored in the memory A0114 as input data A0113. The data before the transformation may be, for example, multiplexed data storing one or more bit streams. Here, the multiplexed data may also be a file having a file format such as ISOBMFF. In addition, the data before the transformation may also be in the form of a plurality of packets generated by dividing the above-mentioned bit stream or file. In the case of transforming data different from the bit stream read from the memory A0114 into a bit stream, the decoding device A0110 may also include a transformation unit not shown in the figure, or the transformation process may be performed by the CPU.

[0248] Decoder A0112 decodes input data A0113 to generate a sound signal A0111 for presenting a prompt to the listener.

[0249] (Another example of an encoding device)

[0250] Figure 5E 1 is a functional block diagram showing the structure of an encoding device A0120 which is another example of the encoding device of the present disclosure. Figure 5E For Figure 5C The same function as the structure of Figure 5C The same reference numerals are used for the components, and description of these components is omitted.

[0251] The coding device A0100 stores the coded data A0103 in the memory A0104 , but the coding device A0120 is different from the coding device A0100 in that it includes a transmission unit A0121 that transmits the coded data A0103 to the outside.

[0252] The transmission unit A0121 transmits a transmission signal A0122 to another device or server based on the coded data A0103 or data in another data format generated by transforming the coded data A0103. The data used to generate the transmission signal A0122 is, for example, the bit stream, multiplexed data, file, or packet described in the coding device A0100.

[0253] (Another example of a decoding device)

[0254] Fig. 5F 1 is a functional block diagram showing the structure of a decoding device A0130 which is another example of the decoding device of the present disclosure. Fig. 5F For Figure 5D The same function as the structure of Figure 5D The same reference numerals are used for the components, and description of these components is omitted.

[0255] The decoding device A0110 reads the input data A0113 from the memory A0114 , whereas the decoding device A0130 is different from the decoding device A0110 in that it includes a receiving unit A0131 that receives the input data A0113 from the outside.

[0256] The receiving unit A0131 receives the received signal A0132 to obtain received data, and outputs input data A0113 to be input to the decoder A0112. The received data may be the same as the input data A0113 to be input to the decoder A0112, or may be data in a data format different from that of the input data A0113. When the received data is data in a data format different from that of the input data A0113, the receiving unit A0131 may convert the received data into the input data A0113, or a conversion unit or CPU (not shown) included in the decoding device A0130 may convert the received data into the input data A0113. The received data is, for example, the bit stream, multiplexed data, file, or packet described in the encoding device A0120.

[0257] (Decoder Functional Description)

[0258] Figure 5G It means as Figure 5D or Fig. 5F A functional block diagram of the structure of a decoder A0200 which is an example of the decoder A0112 in FIG.

[0259] The input data A0113 is a coded bit stream, and includes coded audio data which is a coded audio signal and metadata used in audio processing.

[0260] The spatial information management unit A0201 obtains metadata included in the input data A0113 and parses the metadata. The metadata includes information describing elements that act on the sound and are configured in the sound space. The spatial information management unit A0201 manages the spatial information required for sound processing obtained by parsing the metadata, and provides the spatial information to the rendering unit A0203. In addition, in the present disclosure, the information used in the sound processing is referred to as spatial information, but it may be referred to by other names. Regarding the information used in the sound processing, for example, the information used in the sound processing may be referred to as sound spatial information, or it may be referred to as scene information. In addition, in the case where the information used in the sound processing changes over time, the spatial information input to the rendering unit A0203 may also be referred to as a spatial state, a sound spatial state, a scene state, or the like.

[0261] In addition, the spatial information can also be managed by sound space or by scene. For example, when different rooms are represented as virtual spaces, each room can be managed as a scene of a different sound space, or the spatial information can be managed as a scene that is different depending on the occasion of the representation despite being the same space. In the management of spatial information, an identifier for identifying each piece of spatial information can also be assigned. The data of the spatial information can be included in a bit stream as a form of input data, or the bit stream can include an identifier of the spatial information and the data of the spatial information can be obtained from outside the bit stream. In the case where only the identifier of the spatial information is included in the bit stream, the identifier of the spatial information can also be used during rendering to obtain the data of the spatial information stored in the memory of the sound signal processing device A0001 or in an external server as input data.

[0262] In addition, the information managed by the spatial information management unit A0201 is not limited to the information included in the bitstream. For example, the input data A0113 may include data representing spatial characteristics or structures obtained from a software application or server that provides VR or AR as data not included in the bitstream. In addition, for example, the input data A0113 may include data representing characteristics or positions of a listener or an object as data not included in the bitstream. In addition, the input data A0113 may include information obtained by a sensor possessed by a terminal including a decoding device, or information representing the position of the terminal inferred based on the information obtained by the sensor, as information representing the position of the listener. That is, the spatial information management unit A0201 may also communicate with an external system or server to obtain spatial information and the position of the listener. In addition, the spatial information management unit A0201 may also obtain clock synchronization information from an external system and perform processing synchronized with the clock of the rendering unit A0203. In addition, the space in the above description may be a virtually formed space, i.e., a VR space, or a real space (i.e., a real space) or a virtual space corresponding to the real space, i.e., an AR space or an MR (Mixed Reality) space. In addition, the virtual space may also be referred to as a sound field or a sound space. In addition, the information indicating the position in the above description may be information such as coordinate values ​​indicating the position in the space, information indicating the relative position relative to a prescribed reference position, or information indicating the movement or acceleration of the position in the space.

[0263] The sound data decoder A0202 decodes the encoded sound data contained in the input data A0113 to obtain a sound signal.

[0264] The coded audio data acquired by the stereo sound reproduction system A0000 is, for example, a bit stream coded in a prescribed format such as MPEG-H 3D Audio (ISO / IEC 23008-3). MPEG-H 3D Audio is merely an example of a coding method that can be used when generating coded audio data included in a bit stream, and may include a bit stream and coded audio data coded in other coding methods. For example, the coding method used may be non-reversible coding such as MP3 (MPEG-1 Audio Layer-3), AAC (Advanced Audio Coding), WMA (Windows Media Audio), AC3 (Audio Codec-3), Vorbis, or reversible coding such as ALAC (Apple Lossless Audio Codec), FLAC (Free Lossless Audio Codec), or any coding method other than the above may be used. For example, PCM (pulse code modulation) data may be one type of coded audio data. In this case, the decoding process may be a process of converting an N-bit binary number into a number format (eg, floating point format) that can be processed by the rendering unit A0203, for example, when the number of quantization bits of the PCM data is N.

[0265] The rendering unit A0203 takes the sound signal and spatial information as input, performs audio processing on the sound signal using the spatial information, and outputs the sound signal A0111 after the audio processing.

[0266] Before starting rendering, the spatial information management unit A0201 reads metadata of the input signal, detects rendering items such as objects or sounds specified by the spatial information, and sends them to the rendering unit A0203. After the rendering starts, the spatial information management unit A0201 grasps the changes in the spatial information and the listener's position over time, updates the spatial information and manages it. In addition, the spatial information management unit A0201 sends the updated spatial information to the rendering unit A0203. The rendering unit A0203 generates a sound signal with sound processing added based on the sound signal included in the input data and the spatial information received from the spatial information management unit A0201, and outputs it.

[0267] The spatial information update process and the sound signal output process with added sound processing may be executed by the same thread, or the spatial information management unit A0201 and the rendering unit A0203 may be assigned to separate threads. When the spatial information update process and the sound signal output process with added sound processing are processed by different threads, the thread startup frequency may be set separately, or the processes may be executed in parallel.

[0268] Since the spatial information management unit A0201 and the rendering unit A0203 execute processing in different independent threads, the computing resources can be preferentially allocated to the rendering unit A0203. Therefore, in the case of sound processing in which no delay can be allowed, for example, even in the case of a 1-sample (0.02msec) delay, a small noise can be generated. The sound processing can be safely implemented. At this time, the allocation of computing resources to the spatial information management unit A0201 is restricted. However, the update of spatial information is a lower-frequency process than the output processing of the sound signal (for example, a process such as updating the orientation of the listener's face). Therefore, unlike the output processing of the sound signal, it does not require an instantaneous response, so even if the allocation of computing resources is restricted, it will not have a significant impact on the sound quality brought to the listener.

[0269] The updating of spatial information can be performed periodically at a preset time or period, or when a preset condition is met. In addition, the updating of spatial information can be performed manually by the listener or the manager of the sound space, or it can be performed in response to changes in the external system. For example, when the listener operates the controller and the standing position of his or her avatar is instantly tilted, or the time is instantly moved forward or backward, or when the manager of the virtual space suddenly changes the scene, the thread configured with the spatial information management unit A0201 can be started as a one-time interrupt processing in addition to the regular startup.

[0270] The information update thread that performs the update processing of spatial information is responsible for, for example, updating the position or orientation of the listener's avatar configured in the virtual space based on the position or orientation of the VR goggles worn by the listener, and updating the position of objects moving in the virtual space, etc., provided in a processing thread that is started at a relatively low frequency of about tens of Hz. Such a processing thread with a low occurrence frequency can also be used to perform processing that reflects the properties of direct sound. This is because the frequency of changes in the properties of direct sound is lower than the frequency of occurrence of audio processing frames used for audio output. On the contrary, doing so can relatively reduce the computational load of the processing, and if the information is updated at an unnecessarily fast frequency, there is a risk of generating pulsed noise, so this risk can also be avoided.

[0271] Figure 5H It means as Figure 5D or Fig. 5F A functional block diagram of the structure of a decoder A0210 which is another example of the decoder A0112 in FIG.

[0272] Figure 5H The input data A0113 does not include coded audio data but includes an uncoded audio signal. Figure 5G The input data A0113 includes a bit stream including metadata and an audio signal.

[0273] Space Information Management Department A0211 and Figure 5G The spatial information management unit A0201 is the same as that of the spatial information management unit A0201, so the description is omitted.

[0274] Rendering Department A0213 and Figure 5G The rendering part A0203 is the same, so the description is omitted.

[0275] In addition, in the above description Figure 5H The structure of is called a decoder, but it can also be called an audio processing unit that performs audio processing. In addition, a device including an audio processing unit can also be called an audio processing device instead of a decoding device. In addition, the audio signal processing device A0001 can also be called an audio processing device.

[0276] (Physical Structure of Sound Signal Processing Device)

[0277] Fig.5I is a diagram showing an example of the physical structure of an audio signal processing device. Fig.5I The audio signal processing device may also be a decoding device. In addition, a part of the structure described here may also be equipped in the sound prompting device A0002. In addition, Fig.5I The illustrated sound signal processing device is an example of the sound signal processing device A0001 described above.

[0278] Fig.5I The sound signal processing device includes a processor, a memory, a communication IF, a sensor, and a speaker.

[0279] The processor is, for example, a CPU (Central Processing Unit), a DSP (Digital Signal Processor) or a GPU (Graphics Processing Unit), and the CPU, DSP or GPU may execute a program stored in a memory to implement the audio processing or decoding processing of the present invention. In addition, the processor may also be a dedicated circuit for performing signal processing of sound signals including the audio processing of the present invention.

[0280] The memory is composed of, for example, RAM (Random Access Memory) or ROM (Read Only Memory). The memory may also include a magnetic storage medium such as a hard disk or a semiconductor memory such as an SSD (Solid State Drive). In addition, the memory may also include an internal memory embedded in a CPU or a GPU.

[0281] The communication IF (Interface) is a communication module corresponding to a communication method such as Bluetooth (registered trademark) or WIGI (registered trademark). Fig.5I The acoustic signal processing device shown has a function of communicating with other communication devices via a communication IF, and acquires a bit stream to be decoded, and stores the acquired bit stream in, for example, a memory.

[0282] The communication module is composed of, for example, a signal processing circuit and an antenna corresponding to the communication method. In the above example, Bluetooth (registered trademark) or WIGIG (registered trademark) is cited as an example of the communication method, but it can also correspond to communication methods such as LTE (Long Term Evolution), NR (New Radio) or Wi-Fi (registered trademark). In addition, the communication IF may not be a wireless communication method as described above, but a wired communication method such as Ethernet (registered trademark), USB (Universal Serial Bus), HDMI (registered trademark) (High-Definition Multimedia Interface), etc.

[0283] The sensor performs sensing for estimating the position or orientation of the listener. Specifically, the sensor estimates the position and / or orientation of the listener based on one or more detection results of the position, orientation, movement, speed, angular velocity, or acceleration of a part or the whole of the body such as the head of the listener, and generates position information indicating the position and / or orientation of the listener. In addition, the position information may also be information indicating the position and / or orientation of the listener in real space, or information indicating the displacement of the position and / or orientation of the listener based on the position and / or orientation of the listener at a specified point in time. In addition, the position information may also be information indicating the relative position and / or orientation to an external device having a stereo sound reproduction system or a sensor.

[0284] The sensor may be, for example, a camera or other imaging device or a distance measuring device such as LiDAR (Light Detection And Ranging), and may capture the movement of the listener's head and detect the movement of the listener's head by processing the captured image. In addition, as a sensor, for example, a device that uses wireless position estimation using any frequency band such as millimeter waves may be used.

[0285] in addition, Fig.5I The acoustic signal processing device shown in FIG. 1 may also obtain position information from an external device having a sensor via a communication IF. In this case, the acoustic signal processing device may not include a sensor. Here, the external device is, for example, Figure 5B The sound presentation device A0002 described in , or a stereoscopic image reproduction device worn on the head of the listener, etc. In this case, the sensor is composed of a combination of various sensors such as a gyro sensor and an acceleration sensor.

[0286] For example, as the speed of movement of the listener's head, the sensor can detect the angular velocity of rotation about at least one of the three axes orthogonal to each other in the sound space, or the acceleration of displacement in the direction of displacement about at least one of the three axes.

[0287] For example, as the amount of movement of the listener's head, the sensor can detect the amount of rotation with at least one of the three axes orthogonal to each other in the sound space as the rotation axis, and can also detect the amount of displacement with at least one of the above three axes as the displacement direction. Specifically, the sensor detects 6DoF (position (x, y, z) and angle (yaw, pitch, roll) as the position of the listener. The sensor is composed of a combination of various sensors for detecting movement, such as gyroscope sensors and acceleration sensors.

[0288] In addition, the sensor only needs to detect the position of the listener, and can also be implemented by a camera or a GPS (Global Positioning System) receiver, etc. It is also possible to use position information obtained by self-position estimation using LiDAR (Laser Imaging Detection and Ranging), etc. For example, when the sound signal reproduction system is implemented by a smartphone, the sensor is built into the smartphone.

[0289] In addition, sensors can also include detection Fig.5I A temperature sensor such as a thermocouple for detecting the temperature of the sound signal processing device shown, and a sensor for detecting the remaining amount of a battery included in or connected to the sound signal processing device, etc.

[0290] The speaker has a driving mechanism such as a diaphragm, a magnet or a voice coil, and an amplifier, and presents the sound signal after acoustic processing to the listener as sound. The speaker operates the driving mechanism according to the sound signal amplified by the amplifier (more specifically, a waveform signal representing the waveform of the sound), and the driving mechanism vibrates the diaphragm. In this way, the diaphragm vibrating in response to the sound signal generates sound waves, which propagate in the air and are transmitted to the listener's ears, and the listener perceives the sound.

[0291] In addition, here are Fig.5I The example of the case where the audio signal processing device shown in the figure has a speaker and outputs the audio signal after the audio processing through the speaker is described, but the audio signal prompting mechanism is not limited to the above-mentioned structure. For example, the audio signal after the audio processing can also be output to the external audio prompting device A0002 connected by the communication module. The communication performed by the communication module can be either wired or wireless. In addition, as another example, it can also be Fig.5I The audio signal processing device shown has a terminal for outputting an analog signal of sound, and a cable of an earplug or the like is connected to the terminal to prompt a sound signal from the earplug or the like. In the above case, the earphone, earplug, head-mounted display, neck speaker, wearable speaker, or surround speaker composed of a plurality of fixed speakers worn on the head or a part of the body of the listener as the sound prompting device A0002 reproduces the sound signal.

[0292] (Physical Structure of Encoding Device)

[0293] Figure 5J is a diagram showing an example of the physical structure of an encoding device. Figure 5J The illustrated encoding device is an example of the encoding devices A0100 and A0120 described above.

[0294] Figure 5J The encoding device includes a processor, a memory and a communication IF.

[0295] The processor is, for example, a CPU (Central Processing Unit) or a DSP (Digital Signal Processor), and the encoding process of the present disclosure may be implemented by executing a program stored in a memory by the CPU or GPU. In addition, the processor may also be a dedicated circuit for performing signal processing on a sound signal including the encoding process of the present disclosure.

[0296] The memory is composed of, for example, RAM (Random Access Memory) or ROM (Read Only Memory). The memory may also include a magnetic storage medium such as a hard disk or a semiconductor memory such as an SSD (Solid State Drive). In addition, the memory may also include an internal memory embedded in a CPU or a GPU.

[0297] The communication IF (Interface) is a communication module corresponding to a communication method such as Bluetooth (registered trademark) or WIGI (registered trademark). The encoding device has a function of communicating with other communication devices via the communication IF, and transmits an encoded bit stream.

[0298] The communication module is composed of, for example, a signal processing circuit and an antenna corresponding to the communication method. In the above example, Bluetooth (registered trademark) or WIGIG (registered trademark) is cited as the communication method, but it can also correspond to communication methods such as LTE (LongTerm Evolution), NR (New Radio) or Wi-Fi (registered trademark). In addition, the communication IF may not be a wireless communication method as described above, but a wired communication method such as Ethernet (registered trademark), USB (Universal Serial Bus), HDMI (registered trademark) (High-Definition Multimedia Interface), etc.

[0299] [constitute]

[0300] First, the configuration of the sound signal processing device 100 according to the embodiment will be described. Figure 6 : is a block diagram showing the functional configuration of the sound signal processing device 100 according to the present embodiment.

[0301] The sound signal processing device 100 according to the present embodiment is a device for processing and outputting first sound data representing a first sound caused by an object in a virtual space (sound reproduction space). The sound signal processing device 100 according to the present embodiment is, for example, a device applied to various applications in virtual spaces such as virtual reality or augmented reality (VR / AR).

[0302] Objects in the virtual space are not particularly limited as long as they are objects included in the content displayed on the display unit 30 that displays the content executed in the virtual space (here, images as an example). The object is a moving body, and includes, for example, objects representing animals, plants, artificial objects, natural objects, and the like. Examples of objects representing artificial objects include vehicles, bicycles, and airplanes. In addition, examples of objects representing artificial objects include sporting goods such as baseball bats and tennis rackets; furniture such as tables, chairs, electric fans, and clocks; and buildings such as residential complexes and commercial facilities. In addition, as an example, an object may be at least one of an object that can move within the content and an object that is moved, but is not limited to this. In addition, Figure 3 Right now Figure 4A The electric fan F shown is placed on the ground, and although the electric fan F itself does not move, the blades of the electric fan F move (rotate). Such an electric fan F is also included in the object.

[0303] The first sound is a sound caused by an object. For example, the first sound is a sound produced by an object. More specifically, the first sound is as follows.

[0304] An example of the first sound in this embodiment is aerodynamic sound (wind sound) generated as the object moves in the virtual space, which is the first aerodynamic sound. The wind sound is a sound indicating a vortex train of air generated when the wind W hits the object.

[0305] An example of the first sound related to the present embodiment is aerodynamic sound (second aerodynamic sound) generated in the virtual space by the wind W emitted from the object reaching the ear of the listener L. The second aerodynamic sound is a sound generated, for example, according to the shape of the ear of the listener L when the wind W brought by the electric fan F as the object reaches the listener L. More specifically, the second aerodynamic sound is a sound caused by the wind W generated by the movement of the air caused by the movement of the object.

[0306] The sound signal processing device 100 generates first sound data representing a first sound in a virtual space, and outputs the first sound data to the headphones 20 .

[0307] Next, the earphone 20 will be described.

[0308] The earphone 20 is a device for reproducing the first sound and is a sound output device. More specifically, the earphone 20 reproduces the first sound based on the first sound data output by the sound signal processing device 100. Thus, the listener L can hear the first sound. In addition to the earphone 20, other output channels such as speakers may be used.

[0309] like Figure 6 As shown, the headset 20 includes a head sensor unit 21 and an output unit 22 .

[0310] The head sensor unit 21 senses the position of the listener L in the virtual space determined by horizontal coordinates and vertical height, and outputs second position information indicating the position of the listener L of the first sound in the virtual space to the sound signal processing device 100 .

[0311] The head sensor unit 21 can sense 6DoF (Degrees of Freedom) information of the head of the listener L. For example, the head sensor unit 21 may be an inertial measurement unit (IMU), an accelerometer, a gyroscope, a magnetic sensor, or a combination thereof.

[0312] The output unit 22 is a device that reproduces the sound reaching the listener L in the sound reproduction space. More specifically, the output unit 22 reproduces the first sound based on the first sound data representing the first sound processed by the sound signal processing device 100 and output from the sound signal processing device 100 .

[0313] Next, the display unit 30 will be described.

[0314] The display unit 30 is a display device that displays content (images) including objects in a virtual space. The processing for the display unit 30 to display content will be described later. The display unit 30 is implemented by a display panel such as a liquid crystal panel or an organic EL (ElectroLuminescence) panel.

[0315] Furthermore, Figure 6 The audio signal processing device 100 shown will be described.

[0316] like Figure 6 As shown, the sound signal processing device 100 includes a first acquisition unit 110 , a first calculation unit 120 , a determination unit 130 , a first processing unit 140 , a first output unit 150 , a first reception unit 160 , and a storage unit 170 .

[0317] The first acquisition unit 110 acquires object information and second position information. The object information is information including first position information indicating the position of an object in a virtual space, first sound data indicating a first sound caused by the object, and first identification information indicating a means for processing the first sound data, that is, information related to the object. In addition, the object information may also include shape information indicating the shape of the object. As described above, the second position information is information indicating the position of the listener L in the virtual space. The first acquisition unit 110 may acquire the object information and the second position information from, for example, an input signal, or may acquire the object information and the second position information from other sources. The input signal will be described below.

[0318] The input signal is composed of, for example, spatial information, sensor information, and sound data (sound signal). In addition, the above information and sound data may be included in one input signal, or in a plurality of different signals. The input signal may also include a bit stream composed of sound data and metadata (control information), in which case the metadata may also include spatial information and information for identifying sound data.

[0319] The first position information, the second position information, the shape information, and the availability information described above may also be included in the input signal. More specifically, the first information, the shape information, and the availability information may also be included in the spatial information, and the second information may also be generated based on information obtained from the sensor information. The sensor information may also be obtained from the head sensor unit 201 or from other external devices.

[0320] Spatial information is information about the sound space (three-dimensional sound field) formed by the stereo sound reproduction system, and is composed of information about objects contained in the sound space and information about the listener. Among the objects, there are sound source objects that emit sound and become sound sources, and non-sound objects that do not emit sound. Non-sound objects function as obstacle objects that reflect the sound emitted by sound source objects, but there are also cases where they function as obstacle objects that reflect the sound emitted by other sound source objects as sound source objects. Obstacle objects can also be called reflection objects.

[0321] The information given to both the sound source object and the non-sound emitting object includes position information, shape information, and a volume attenuation rate when the object reflects sound.

[0322] The position information is represented by the coordinate values ​​of three axes, such as the X-axis, Y-axis, and Z-axis in Euclidean space, but it does not necessarily have to be three-dimensional information. The position information may also be two-dimensional information represented by the coordinate values ​​of two axes, such as the X-axis and the Y-axis. The position information of the object is determined by the representative position of the shape represented by the grid or voxel.

[0323] The shape information may also include information about the material of the surface.

[0324] The attenuation rate can be expressed by a real number below 1 or above 0, or by a negative decibel value. Since the volume is not amplified by reflection in real space, the attenuation rate is set to a negative decibel value, but for example, in order to express the horror of an unreal space, an attenuation rate of 1 or above, that is, a positive decibel value, can be deliberately set. In addition, the attenuation rate can be set to a different value for each frequency band constituting a plurality of frequency bands, or a value can be set independently for each frequency band. In addition, when the attenuation rate is set for each type of material on the surface of the object, the corresponding attenuation rate value can also be used based on information related to the material of the surface.

[0325] In addition, the information assigned to the sound source object and the non-sound-generating object may also include information indicating whether the object is a living thing, or information indicating whether the object is a moving object, etc. In the case where the object is a moving object, the position information may also move over time, and the changed position information or the amount of change is transmitted to the rendering units A0203 and A0213.

[0326] The information related to the sound source object includes not only the information given to the sound source object and the non-sound-generating object, but also the sound data and the information required to radiate the sound data into the sound space. The sound data is data that represents the sound perceived by the listener, such as information related to the frequency and strength of the sound. The sound data is typically a PCM signal, but it can also be data compressed using an encoding method such as MP3. In this case, at least the signal needs to be transmitted to the generator (in the Fig.40 Since the signal is decoded before the generation unit 907 described later in the figure, a decoding unit not shown in the figure may be included in the rendering unit A0203 and A0213. Alternatively, the signal may be decoded by the sound data decoder A0202.

[0327] It is sufficient to set at least one sound data for one sound source object, or multiple sound data may be set. In addition, identification information for identifying each sound data may be given, and the identification information of the sound data may be stored as metadata as information related to the sound source object.

[0328] The information required for radiating sound data into the sound space may include, for example, information on a reference volume used as a reference when reproducing sound data, information related to the position of the sound source object, information related to the orientation of the sound source object, and information related to the directionality of the sound emitted by the sound source object.

[0329] The reference volume information is, for example, the effective value of the amplitude value of the sound data at the sound source position when the sound data is radiated into the sound space, and may also be expressed as a decibel (db) value with a floating decimal point. For example, when the reference volume is 0db, the reference volume information may also indicate that the sound is radiated to the sound space from the position indicated by the information related to the above position without increasing or decreasing the volume of the signal level represented by the sound data. In the case of -6db, the reference volume information may also indicate that the sound is radiated to the sound space from the position indicated by the information related to the above position with the volume of the signal level represented by the sound data reduced to approximately half. The reference volume information may also be assigned to one sound data or to a plurality of sound data together.

[0330] The volume information included in the information required to radiate the sound data into the sound space may include, for example, information indicating the time series changes of the volume of the sound source. For example, when the sound space is a virtual conference room and the sound source is a speaker, the volume changes intermittently in a short period of time. If it is expressed more simply, it can be said that the sound part and the silent part are produced alternately. In addition, when the sound space is a concert hall and the sound source is a performer, the volume is maintained for a certain length of time. In addition, when the sound space is a battlefield and the sound source is an explosive, the volume of the explosion only increases for a moment, and then it is silent and continues. In this way, the information on the volume of the sound source includes not only the information on the size of the sound, but also the information on the changes in the size of the sound, and such information can also be set as information indicating the nature of the sound data.

[0331] Here, the information on the change in the volume of the sound may also be data representing the frequency characteristics in a time series. The information on the change in the volume of the sound may also be data representing the duration of the sound interval. The information on the change in the volume of the sound may also be data representing the duration of the sound interval and the time length of the silent interval. The information on the change in the volume of the sound may also be data listing the duration of the amplitude of the sound signal as constant (regarded as approximately constant) and the amplitude value of the signal during the duration in a plurality of time series. The information on the change in the volume of the sound may also be data listing the duration of the frequency characteristics of the sound signal as constant and the frequency characteristics during the duration in a plurality of time series. The information on the change in the volume of the sound may also be data representing the approximate shape of a spectrogram as data. In addition, the volume as a reference for the above-mentioned frequency characteristics may also be set as the above-mentioned reference volume. The information indicating the reference volume and the information indicating the properties of the sound data are used not only for calculating the volume of the direct sound or the reflected sound to be perceived by the listener, but also for selection processing for selecting whether to make the listener perceive the sound.

[0332] The information related to the orientation is typically expressed by yaw, pitch, and roll. Alternatively, the rotation of roll may be omitted and expressed by azimuth (yaw) and elevation (pitch). The orientation information may also change over time and, if changed, is transmitted to the rendering units A0203 and A0213.

[0333] Information related to the listener is information related to the position information and orientation of the listener in the sound space. The position information is represented by the position of the XYZ axis of the Euclidean space, but it does not necessarily have to be three-dimensional information, and it can also be two-dimensional information. Information related to the orientation is typically represented by yaw, pitch, and roll. Alternatively, the rotation of roll can be omitted and represented by azimuth (yaw) and elevation (pitch). The position information and orientation information can also change over time, and when they change, they are transmitted to the rendering units A0203 and A0213.

[0334] The sensor information is information including the amount of rotation or displacement detected by the sensor worn by the listener and the position and orientation of the listener. The sensor information is transmitted to the rendering units A0203 and A0213, and the rendering units A0203 and A0213 update the information of the position and orientation of the listener based on the sensor information. For example, the sensor information can also use the position information obtained by the portable terminal using GPS, camera or LiDAR (Laser Imaging Detection and Ranging) to estimate its own position. In addition, information obtained from the outside via the communication module other than the sensor can also be detected as sensor information. Information indicating the temperature of the sound signal processing device 100 and information indicating the remaining battery level can also be obtained from the sensor as sensor information. Information indicating the computing resources (CPU capacity, memory resources, PC performance) of the sound signal processing device 100 or the sound prompt device A0002 can also be obtained in real time as sensor information.

[0335] In this embodiment, the first acquisition unit 110 acquires the object information from the storage unit 170, but the present invention is not limited thereto, and the object information may be acquired from a device other than the sound signal processing device 100 (for example, a server device 10 such as a cloud server). In addition, the first acquisition unit 110 acquires the second position information from the earphone 20 (more specifically, the head sensor unit 21), but the present invention is not limited thereto.

[0336] Here, information included in the object information is described.

[0337] First, the first position information will be described.

[0338] As described above, the object in the virtual space is included in the content (image) displayed by the display unit 30, and may be at least one of an object that can move within the content and an object that can be moved. Figure 1 Bat B shown.

[0339] The first position information is information indicating where the bat B in the virtual space is located at a certain point in time. In addition, in the virtual space, the bat B may move due to the swinging of the bat B. Therefore, the first acquisition unit 110 continuously acquires the first position information. The first acquisition unit 110 acquires the first position information every time the spatial information management units A0201 and A0211 execute the updating of the spatial information, for example.

[0340] Next, the first audio data representing the first audio will be described.

[0341] The audio data including the first audio data described in this specification may be an audio signal such as PCM (Pulse Code Modulation) data, but is not limited thereto, and may be any information for indicating the properties of audio.

[0342] For example, if the sound signal is a noise signal with a volume of X decibels, the sound data related to the sound data may be the PCM data representing the sound signal itself, or may be data consisting of information indicating that the component is a noise signal and information indicating that the volume is X decibels. For another example, if the sound signal is a noise signal with a Peak / Dip of a frequency component having a predetermined characteristic, the sound data related to the sound data may be the PCM data representing the sound signal itself, or may be data consisting of information indicating that the component is a noise signal and information indicating the Peak / Dip of the frequency component.

[0343] In this specification, a sound signal based on sound data refers to PCM data representing the sound data.

[0344] Next, the first identification information will be described.

[0345] The first identification information is information indicating the means for processing the first sound data. That is, in the present embodiment, a first processing means and a second processing means are provided as the means for processing the first sound data. The first processing means and the second processing means are both means for processing the volume of the first sound represented by the first sound data, and process the first sound data in mutually different forms. The first identification information is information indicating whether the means for processing the first sound data is the first processing means, the second processing means, or both the first processing means and the second processing means.

[0346] What processing means the first identification information indicates is predetermined according to the object indicated by the first identification information. For example, what processing means the first identification information indicates is predetermined by the creator of the content (video) displayed on the display unit 30.

[0347] Next, shape information will be described.

[0348] Shape information is information indicating the shape of an object (e.g., bat B) in a virtual space. Shape information indicates the shape of an object, and more specifically, indicates a three-dimensional shape as a rigid body of the object. The shape of the object is represented by, for example, a sphere, a cuboid, a cube, a polyhedron, a cone, a pyramid, a cylinder, a prism, or a combination thereof. In addition, shape information may be represented as mesh data, or a collection of multiple faces composed of voxels, a three-dimensional point group, or vertices having three-dimensional coordinates, for example.

[0349] Furthermore, the first position information includes object identification information for identifying the object, the first identification information also includes object identification information for identifying the object, and the shape information also includes object identification information for identifying the object.

[0350] Therefore, even if the first acquisition unit 110 acquires the first position information, the first sound data, the first identification information, and the shape information separately, the objects indicated by the first position information, the first sound data, the first identification information, and the shape information are identified by referring to the object identification information included in each of the first position information, the first sound data, the first identification information, and the shape information. For example, it can be easily identified that the objects indicated by the first position information, the first sound data, the first identification information, and the shape information are the same bat B. That is, with respect to each of the first position information, the first sound data, the first identification information, and the shape information acquired by the first acquisition unit 110, by referring to the four pieces of object identification information, it is clear that the first position information, the first sound data, the first identification information, and the shape information are information related to the bat B. Therefore, the first position information, the first sound data, the first identification information, and the shape information are associated as information indicating the bat B.

[0351] Next, the second position information will be described.

[0352] The listener L can move in the virtual space. The second position information is information indicating where the listener L in the virtual space is at a certain point in time. In addition, since the listener L can move in the virtual space, the first acquisition unit 110 continuously acquires the second position information. The first acquisition unit 110 acquires the first position information every time the spatial information management units A0201 and A0211 execute an update of the spatial information, for example.

[0353] In addition, the first position information, the first sound data, the first identification information, the shape information, etc. mentioned above may also be included in the metadata, control information, or header information included in the input signal. In the case where the first sound data is a sound signal (PCM data), information for identifying the sound signal may be included in the metadata, control information, or header information, and the sound signal may be included outside the metadata, control information, or header information. That is, the sound signal processing device 100 (more specifically, the first acquisition unit 110) may also acquire the metadata, control information, or header information included in the input signal, and perform sound processing based on the metadata, control information, or header information. In addition, the sound signal processing device 100 (more specifically, the first acquisition unit 110) only needs to acquire the first position information, the first sound data, the first identification information, and the shape information, and the acquisition location is not limited to the input signal. The first sound data and the metadata may be stored in one input signal or in a plurality of input signals.

[0354] In addition, a sound signal other than the first sound data may be stored in the input signal as audio content information. The audio content information may also be subjected to encoding processing such as MPEG-H 3D Audio (ISO / IEC 23008-3) (hereinafter referred to as MPEG-H 3D Audio). In addition, the technology used in the encoding processing is not limited to MPEG-H 3D Audio, and other well-known technologies may also be used. In addition, the above-mentioned first position information, first sound data, first identification information, shape information and other information may also be used as encoding processing objects.

[0355] That is, the sound signal processing device 100 obtains the sound signal and metadata included in the encoded bit stream. In the sound signal processing device 100, the audio content information is obtained and decoded. In the present embodiment, the sound signal processing device 100 functions as a decoder included in the decoding device, and more specifically, functions as the rendering units A0203 and A0213 included in the decoder. In addition, the term "audio content information" in the present disclosure can be replaced with the sound signal itself, or information including the first position information, the first sound data, the first identification information, and the shape information in terms of technical content.

[0356] In addition, the second position information may also be subjected to encoding processing. That is, the first acquisition unit 110 acquires the second position information and decodes it.

[0357] The first acquisition unit 110 outputs the acquired object information and second position information to the first calculation unit 120 and the determination unit 130 .

[0358] The first calculation unit 120 calculates the distance between the object (for example, the bat B) and the listener L based on the first position information included in the object information obtained by the first acquisition unit 110 and the second position information obtained. As described above, the first acquisition unit 110 obtains the first position information and the second position information in the virtual space each time the spatial information is updated by the spatial information management units A0201 and A0211. The first calculation unit 120 calculates the distance between the object and the listener L in the virtual space based on the plurality of first position information and the plurality of second position information obtained each time the spatial information is updated. The first calculation unit 120 outputs the calculated distance between the object and the listener L to the determination unit 130.

[0359] The decision unit 130 decides a processing means for processing the first sound data between the first processing means and the second processing means based on the first identification information included in the object information acquired by the first acquisition unit 110. The first processing means is a processing means for performing a process corresponding to the volume of the sound calculated by the first calculation unit 120. The second processing means is a processing means for performing a process corresponding to the volume of the sound calculated by the first calculation unit 120 in a different form from the first processing means.

[0360] As described above, the first identification information indicates the processing means for the first sound data, and the decision unit 130 determines the processing means for processing the first sound data according to the processing means indicated by the first identification information. For example, when the first processing means is indicated as the processing means for the first sound data in the first identification information, the decision unit 130 determines that the processing means for processing the first sound data is the first processing means.

[0361] The first processing unit 140 processes the first sound data by the processing means determined by the determination unit 130. For example, when the determination unit 130 determines that the processing means for processing the first sound data is the first processing means, the first processing unit 140 processes the first sound data by the first processing means.

[0362] The first output unit 150 outputs the first audio data processed by the first processing unit 140. Here, the first output unit 150 outputs the first audio data to the earphone 20. Thus, the earphone 20 can reproduce the first audio represented by the outputted first audio data.

[0363] The first receiving unit 160 receives an operation from a user of the sound signal processing device 100 (for example, a creator of content executed in a virtual space). The first receiving unit 160 is specifically implemented by a hardware button, but may also be implemented by a touch panel or the like. The operation may include an operation of specifying a file name, and the file may be a file in which object information including the first position information, the first sound data, the first identification information, and the shape information is formatted according to a prescribed rule. The first receiving unit 160 may also receive the object information by deformatting the file. This is not limited to the first receiving unit 160, and the same applies to the receiving unit 41, the second receiving unit 51, and the third receiving unit 61 described later.

[0364] The storage unit 170 is a storage device that stores computer programs and the like executed by the first acquisition unit 110 , the first calculation unit 120 , the determination unit 130 , the first processing unit 140 , the first output unit 150 , and the first reception unit 160 , and object information.

[0365] Here, the shape information related to this embodiment is described again. The shape information is information used to generate an image of an object in a virtual space, and is also information indicating the shape of the object (bat B). That is, the shape information is information used to generate content (image) displayed on the display unit 30.

[0366] The first acquisition unit 110 also outputs the acquired shape information to the display unit 30. The display unit 30 acquires the shape information output by the first acquisition unit 110. The display unit 30 also acquires attribute information indicating attributes (color, etc.) other than the shape of the object (bat B) in the virtual space. The display unit 30 may directly acquire the attribute information from a device (server device 10) other than the sound signal processing device 100, or may acquire it from the sound signal processing device 100. The display unit 30 generates content (image) based on the acquired shape information and attribute information, and displays it.

[0367] In addition, the first processing means and the second processing means are described again.

[0368] The first processing means is a processing means for processing the first audio data so that the volume is attenuated inversely proportional to the increase in the distance calculated by the first calculation unit 120 .

[0369] If the distance is D, and the volume of the first sound processed by the first processing means is V1, V1 is expressed by (Formula 1).

[0370] V1∝(1 / D)…(Formula 1)

[0371] In addition, the second processing means is a processing means for processing the first sound data in a manner that increases or decreases the volume in a different form from the first processing means as the distance calculated by the first calculation unit 120 increases. As an example, the second processing means is a processing means for processing the first sound data in a manner that causes the volume to decay as the distance is raised to the power of x (x≠1). At this time, if the volume of the first sound processed by the second processing means is V2, V2 is represented by (Formula 2). In addition, "^" in (Formula 2) is an exponentiation operator.

[0372] V2∝(1 / D)^x…(Formula 2)

[0373] In addition, in the present embodiment, the first identification information is also information indicating that the processing means for the first sound data is the second processing means, and the value of x. More specifically, when the first sound is a pneumatic sound (wind sound) generated as the object moves, the first identification information indicates that the processing means for the first sound data is the second processing means, and the above-mentioned x is α, α is a real number, and α satisfies (Formula 3).

[0374] α>1…(Formula 3)

[0375] Furthermore, first accepting unit 160 accepts an operation of designating a value of α from a user of acoustic signal processing device 100 . As a result, for example, x indicated by the first identification information included in the object information stored in storage unit 170 becomes α. That is, x indicated by the first identification information is updated to α.

[0376] Furthermore, it is described here that the first sound data is processed according to (Formula 1) as the first processing means and according to (Formula 2) as the second processing means, but it is not limited to this. For example, a function f with the distance d calculated by the first calculation unit 120 as the independent variable can also be used, and V3 calculated as V3 = f(d^r) (r is a real number greater than 1) can be used as the volume. In many cases, a monotonically decreasing function such as an inverse proportional function is used in the function f, but in view of the balance between the sense of reality in the real space and the special effects (such as immersion and entertainment) in the virtual space, a function different from the monotonically decreasing function can also be used. The function to be used is set in advance by the system designer. In addition, multiple functions can be prepared in advance, and the multiple functions can be switched and used based on parameters such as object information or the position of the object, the state or position of the listener, etc.

[0377] Furthermore, instead of using the function f, a table structure having a volume V4 corresponding to the distance d can be used. As for what kind of table to make, the system designer can design it with a high degree of freedom in view of the balance between the sense of reality in the real space and the special effects (such as immersion and entertainment) in the virtual space. The table to be used is set in advance by the system designer. In addition, multiple tables can be prepared in advance, and the multiple functions can be switched and used based on parameters such as object information or the position of the object, the state or position of the listener, etc.

[0378] Hereinafter, operation example 1 of the sound signal processing method performed by the sound signal processing device 100 will be described.

[0379] [Action Example 1]

[0380] Figure 7 This is a flowchart of operation example 1 of the sound signal processing device 100 according to the present embodiment. Figure 8 1 is a diagram showing a bat B and a listener L as objects related to Action Example 1.

[0381] In the present operation example, the first sound is a sound caused by the target bat B, and is a first pneumatic sound (wind sound) generated as the bat B moves.

[0382] like Figure 7 As shown, first, the first accepting unit 160 accepts an operation of designating a value of α as an example of x indicated by the first identification information ( S10 ). This step S10 corresponds to a first accepting step.

[0383] Thus, x indicated by the first identification information included in the object information stored in the storage unit 170 becomes α.

[0384] Furthermore, the first acquisition unit 110 acquires the object information including the first position information, the first audio data, and the first identification information, and the second position information (S20). This step S20 corresponds to the first acquisition step.

[0385] Next, the first calculation unit 120 calculates the distance D between the object and the listener L based on the first position information included in the object information acquired by the first acquisition unit 110 and the second position information acquired by the first acquisition unit 110 (S30). That is, here, the first calculation unit 120 calculates the distance D between the bat B and the listener L. This step S30 corresponds to the first calculation step.

[0386] Next, the decision unit 130 decides a processing means for processing the first audio data between the first processing means and the second processing means based on the first identification information included in the object information acquired by the first acquisition unit 110 (S40). This step S40 corresponds to a decision step.

[0387] Next, the first processing unit 140 processes the first audio data by the processing means determined by the determination unit 130 (S50). This step S50 corresponds to the first processing step.

[0388] For example, when the processing means is determined to be the first processing means in step S40, the first processing unit 140 processes the first sound data by the first processing means. In addition, for example, when the processing means is determined to be the second processing means in step S40, the first processing unit 140 processes the first sound data by the second processing means. Here, it is assumed that the processing means is determined to be the second processing means in step S40, and the second processing means is a processing means for processing the first sound data in a manner that the volume is attenuated by the power of x of the distance D. According to step S10, since x is α, if the volume of the first sound processed by the second processing means is V2, V2 is expressed by (Formula 4).

[0389] V2∝(1 / D)^α…(Formula 4)

[0390] Also, for example, α is 2.

[0391] The first output unit 150 outputs the first audio data processed by the first processing unit 140 (S60). This step S60 corresponds to a first output step.

[0392] In the present embodiment, since the processing means can change the volume of the first sound according to the first identification information, the first sound in the virtual space listened to by the listener L becomes a sound similar to the first sound in the real space listened to by the listener L, more specifically, a sound that reproduces the first sound in the real space. Therefore, the listener L is less likely to feel awkward and the listener L can obtain a sense of presence.

[0393] Furthermore, in the present embodiment, the first sound is the first pneumatic sound (wind sound). In addition, based on the first identification information, a first processing means is used to process the first sound data in a manner that causes the volume to decay inversely proportional to the increase in distance D, or a second processing means is used to increase or decrease the volume in a manner different from the first processing means as the distance D increases. Therefore, the first sound (wind sound) in the virtual space listened to by the listener L becomes a sound that is more similar to the first sound (wind sound) in the real space listened to by the listener L. Therefore, the listener L is less likely to feel awkward, and the listener L can get a better sense of presence.

[0394] (Variation 1 of the embodiment)

[0395] Hereinafter, Modification 1 of the embodiment will be described. Hereinafter, the description will be mainly focused on the differences from the embodiment, and the description of the common points will be omitted or simplified.

[0396] [constitute]

[0397] First, the configuration of an acoustic signal processing device 100 a according to Modification 1 of the present embodiment will be described. Fig. 9 : is a block diagram showing the functional configuration of an audio signal processing device 100 a according to this modification.

[0398] The sound signal processing device 100a according to this modification can present in a virtual space a personal space corresponding to the listener L and a sense of distance between the listener L and other persons that cannot be represented by physical distance alone. The sound signal processing device 100a has the same configuration as the sound signal processing device 100 according to the embodiment, except that the first processing unit 140a is replaced with the first processing unit 140.

[0399] In this modification, the objects in the virtual space are other persons other than the listener L. Fig.10 4 other persons A1 to D1 and a listener L according to this modification example are shown in FIG. Fig.10 In the example, four other persons A1 to D1 are classified into four categories: close distance, individual distance, social distance, and public distance of personal space.

[0400] For example, it is assumed that the intimacy with the listener L decreases in the order of other person A1, other person B1, other person C1, and other person D1. And the listener L can allow other person A1 to approach to a close distance, can allow other person B1 to approach to an individual distance, can allow other person C1 to approach to a social distance, and can allow other person D1 to approach to a public distance.

[0401] Furthermore, in this variation, when the object is another person A1, the first sound is the sound of another person A1. Similarly, when the object is another person B1, the first sound is the sound of another person B1, when the object is another person C1, the first sound is the sound of another person C1, and when the object is another person D1, the first sound is the sound of another person D1.

[0402] The first acquisition unit 110 of the sound signal processing device 100a acquires object information corresponding to other persons corresponding to the object and second position information. There are object information corresponding to four other persons A1 to D1, respectively, and the object information corresponding to the other person A1 is sometimes recorded as object information A1. Similarly, the object information corresponding to the other person B1 is sometimes recorded as object information B1, the object information corresponding to the other person CB1 is sometimes recorded as object information C1, and the object information corresponding to the other person D1 is sometimes recorded as object information D1. In addition, when it is not necessary to identify the object information A1 to D1 separately, they are sometimes simply recorded as object information.

[0403] The first processing unit 140a according to this modification processes the first audio data by the processing means determined by the determination unit 130. Here, the first, second, and third processing methods performed by the first processing unit 140a will be described.

[0404] <Example 1>

[0405] First, the processing method of the first example is described.

[0406] In the first example, the second processing means according to this variation is a processing means for processing the first sound data so that the volume becomes a predetermined value when the distance D calculated by the first calculation unit 120 is within a predetermined threshold. The predetermined threshold is a value corresponding to the personal space.

[0407] That is, the second processing means processes the first sound data so that the volume heard by the listener L becomes a predetermined value according to the distance D. The predetermined value is, for example, a value VH indicating a larger volume and a value VL indicating a smaller volume than the VH. More specifically, VH is a volume that is so loud that the listener L feels disgusted when another person approaches the listener L, and VL is a volume that the listener L feels that another person is approaching the listener L.

[0408] First, in this variation, the second processing means is described for a case where the object is another person A1 or the like classified as a close distance category. The listener L is close to the other person A1 and is allowed to approach to a close distance (less than 45 cm). Therefore, the second processing means is a processing means for processing the first sound data in such a manner that the volume of the sound of the other person A1 becomes VL when the distance D is less than 45 cm, and in such a manner that the volume of the sound of the other person A1 attenuates as the distance D increases when the distance D is greater than 45 cm.

[0409] Next, the second processing means is described for the case where the object is other person B1 and is classified as the category of individual distance. The listener L is somewhat close to the other person B1, and is allowed to approach to the individual distance (greater than 45 cm and less than 120 cm). Therefore, the second processing means is a processing means for processing the first sound data in such a way that the volume of the sound of the other person B1 becomes VH when the distance D is less than 45 cm, and in such a way that the volume of the sound of the other person B1 becomes VL when the distance D is greater than 45 cm and less than 120 cm, and in such a way that the volume of the sound of the other person B1 attenuates as the distance D increases when the distance D is greater than 120 cm.

[0410] Next, the second processing means is described for the case where the object is other person C1 or the like classified as the social distance. The listener L is not close to the other person C1, and is allowed to approach to the social distance (greater than 120 cm and less than 350 cm). Therefore, the second processing means is a processing means for processing the first sound data in such a way that the volume of the sound of the other person C1 becomes VH when the distance D is less than 120 cm, and in such a way that the volume of the sound of the other person C1 becomes VL when the distance D is greater than 120 cm and less than 350 cm, and in such a way that the volume of the sound of the other person C1 attenuates as the distance D increases when the distance D is greater than 350 cm.

[0411] Next, the second processing means is described when the target is another person D1 or the like classified as a public distance. The listener L is not close to the other person D1 and is allowed to approach the public distance (350 cm). Therefore, the second processing means is a processing means for processing the first sound data in such a manner that the volume of the sound of the other person D1 becomes VH when the distance D is less than 350 cm, and in such a manner that the volume of the sound of the other person D1 attenuates as the distance D increases when the distance D is greater than 350 cm.

[0412] In addition, the category into which each of the other persons A1 to D1 is classified in the personal space can be indicated by the first identification information included in the corresponding object information A1 to D1.

[0413] In addition, as described above, the prescribed threshold is a value corresponding to the personal space, but for example, the first accepting unit 160 may accept an operation from the user to designate that the prescribed threshold is a first designated value, and the first designated value designated by the accepted operation may become the prescribed threshold.

[0414] <Example 2>

[0415] Next, the processing method of the second example is described.

[0416] In the second example, before the processing of the first processing unit 140a is performed, the decision unit 130 determines whether to execute the first processing means based on the first identification information obtained by the first obtaining unit 110, and determines to execute the second processing means regardless of whether the first processing means is executed. In the second example, the first identification information is information indicating whether to execute the first processing means, and is not information indicating whether to execute the second processing means.

[0417] That is, the first processing unit 140a performs the first processing means on the first sound data when the determination unit 130 determines to perform the first processing means, and does not perform the first processing means on the first sound data when the determination unit 130 determines not to perform the first processing means. Furthermore, the first processing unit 140a performs the second processing means on the first sound data regardless of whether the first processing means is performed on the first sound data.

[0418] In the second example, the second processing means according to this variation is also a processing means for processing the first sound data so that the volume becomes a predetermined value when the distance D calculated by the first calculation unit 120 is within a predetermined threshold. The predetermined threshold is a value corresponding to the personal space.

[0419] Hereinafter, an example will be described in which both the first processing means and the second processing means are executed on the first audio data.

[0420] First, by executing the first processing means, the first sound data is processed in such a way that the sound is attenuated inversely proportional to the increase in the distance D, regardless of whether the distance D between the object (for example, the other person B1) and the listener L is within a predetermined threshold. Then, the second processing means is executed on the first sound data processed by the first processing means. The second processing means related to the second example is a processing means for processing the first sound data in such a way that the volume of the sound of the other person B1 becomes VH when the distance D is 45 cm, and in such a way that the volume of the sound of the other person B1 becomes VL when the distance D is 120 cm.

[0421] Therefore, the first sound data after executing the first processing means and the second processing means indicates that the sound attenuates inversely as the distance D increases, and when the distance D is 45 cm, the volume of the sound of the other person B1 is VH, and when the distance D is 120 cm, the volume of the sound of the other person B1 is VL.

[0422] Furthermore, the second processing means according to the second example is a processing means for processing the first voice data so that the volume of the voice of the other person A1 becomes VL when the distance D is 45 cm when the target is the other person A1.

[0423] In addition, the second processing means related to the second example is a processing means for processing the first sound data in such a manner that when the object is another person C1, the volume of the other person C1's voice becomes VH when the distance D is 120 cm, and in such a manner that when the distance D is 350 cm, the volume of the other person C1's voice becomes VL.

[0424] Furthermore, the second processing means according to the second example is a processing means for processing the first voice data so that the volume of the voice of the other person D1 becomes VH when the distance D is 350 cm when the target is the other person D1.

[0425] <Example 3>

[0426] Next, the processing method of the third example is described.

[0427] In the third example, the second processing means related to this variation is a processing means for processing the first sound data in such a manner that the volume is attenuated by the xth power (x≠1) of the distance D. Furthermore, the higher the intimacy between the listener L and the other person, the smaller the value of x related to the other person.

[0428] For example, when the object is other person A1 or the like classified as a close distance category, the value of x is 0.9, and when the object is other person B1 or the like classified as an individual distance category, the value of x is 1.5. In addition, for example, when the object is other person C1 or the like classified as a social distance category, the value of x is 2.0, and when the object is other person D1 or the like classified as a public distance category, the value of x is 3.0.

[0429] In the first to third examples above, various variables are set to make the sound louder so that people feel disgusted when others who are not close to them approach, and vice versa, because people do not want to hear disgusting sounds, they can also be set to a smaller volume. That is, while the first processing means is to reproduce physical phenomena, the second processing means can also be used to present the increase or decrease of volume based on psychological distance that cannot be expressed by physical distance.

[0430] Next, operation example 2 of the sound signal processing method performed by the sound signal processing device 100 a will be described.

[0431] [Action Example 2]

[0432] Fig.11 This is a flowchart of Operation Example 2 of the sound signal processing device 100 a according to this modification.

[0433] like Fig.11 As shown, first, the first accepting unit 160 accepts an operation from the user to designate that the predetermined threshold is a first designated value (S11). As an example, the predetermined threshold is a value corresponding to the personal space. In the case of the third example described above, step S11 may not be performed.

[0434] Furthermore, the first acquisition unit 110 acquires object information including the first position information, the first audio data, and the first identification information, and the second position information (S20). The object information acquired in step S20 is at least one of the four pieces of object information A1 to D1.

[0435] Next, the first calculation unit 120 calculates the distance D between the object (other person) and the listener L based on the first position information included in the object information acquired by the first acquisition unit 110 and the second position information acquired by the first acquisition unit 110 ( S30 ).

[0436] Next, the decision unit 130 decides a processing means for processing the first audio data between the first processing means and the second processing means based on the first identification information included in the object information acquired by the first acquisition unit 110 ( S40 ).

[0437] Next, the first processing unit 140a processes the first audio data by the processing means determined by the determination unit 130 (S50). As described above, the first audio data is processed by using any of the first to third methods.

[0438] The first output unit 150 outputs the first audio data processed by the first processing unit 140a (S60).

[0439] For example, in this variation, when another person D1 who is not close to the listener L approaches to 350 cm, the listener L listens to the voice of the other person D1 at a volume of VH that the listener L feels disgusted with. On the other hand, when another person A1 who is close to the listener L approaches to 45 cm, the listener L listens to the voice of the other person A1 at a volume of VL that the listener L feels close to. That is, the volume of the voice of the other person D1 who is not close is controlled to a level that is disgusting, and the volume of the voice of the other person A1 who is close is controlled to a level that can be heard. Therefore, in this variation, the personal space corresponding to the listener L and the sense of distance between the listener L and other people that cannot be expressed only by physical distance can be presented in the virtual space.

[0440] In summary, in this variation, in the first processing step, the second processing means can be used to process the first sound data in such a way that the volume becomes a predetermined value when the distance D is within a predetermined threshold value. Furthermore, in the first processing step, the first sound data can be processed using a predetermined threshold value that is a value corresponding to the personal space. Therefore, the sound signal processing device 100a related to this variation can present the personal space corresponding to the listener L and the sense of distance between the listener L and other persons that cannot be expressed only by the physical distance in the virtual space by executing the second processing means whose predetermined value is different for each other person as the object.

[0441] (Variation 2 of the embodiment)

[0442] Hereinafter, Modification 2 of the embodiment will be described. Hereinafter, the description will be mainly focused on the differences from the embodiment, and the description of the common points will be omitted or simplified.

[0443] [constitute]

[0444] First, the configuration of an acoustic signal processing device 100 b according to Modification 2 of the present embodiment will be described. Fig.12 : is a block diagram showing the functional configuration of an acoustic signal processing device 100b according to this modification.

[0445] The sound signal processing device 100b according to this modification can process a plurality of sound data. The sound signal processing device 100b has the same configuration as the sound signal processing device 100 according to the embodiment, except that the first processing unit 140 is replaced with a first processing unit 140b.

[0446] Fig.13 2 is a diagram showing an object and a plurality of sounds related to this modification. In this modification, the object in the virtual space is an ambulance A. The plurality of sounds include three sounds, more specifically, two first sounds and one second sound.

[0447] The two first sounds and the one second sound are sounds caused by the target ambulance A. In order to distinguish the two first sounds, they are recorded as first sound A2 and first sound B2.

[0448] The first sound A2 is the aerodynamic sound (first aerodynamic sound) generated as the object (ambulance A) moves, that is, the sound of wind. The first sound B2 is the aerodynamic sound generated by the wind W emitted from the object (ambulance A) reaching the ears of the listener L, that is, the second aerodynamic sound. That is, the wind W emitted from the object is like Fig.13 The figure also includes the wind generated by the movement of the object (ambulance A) such as a moving body.

[0449] The second sound is a sound different from the two first sounds, and more specifically, is a whistle sound emitted from the ambulance A. Thus, the ambulance A is an object that generates a plurality of sounds including the wind whistling sound (the first pneumatic sound), the second pneumatic sound, and the whistle sound.

[0450] In this variation, the object information acquired by the first acquisition unit 110 includes the first position information, the first sound data indicating the first sound, the first identification information, the second sound data indicating the second sound, and the second identification information indicating the processing means for the second sound data. In addition, the first sound data includes the first sound data A2 indicating the first sound A2 and the first sound data B2 indicating the first sound B2. In addition, the first identification information includes the first identification information A2 indicating the processing means for the first sound data A2 and the first identification information B2 indicating the processing means for the first sound data B2.

[0451] Furthermore, in the object information related to the present modification, the first sound data B2 indicating the first sound B2 (second pneumatic sound) generated by the object at the position associated with the position of the listener L is associated with the position of the object (ambulance A) indicated by the first position information. Further, in the real space, the second pneumatic sound is generated at the ear of the listener L, but here the position of the ambulance A in the virtual space is regarded as the position of the sound source.

[0452] That is, the ambulance A is an object associated with a plurality of sound data including the first sound data and the second sound data (here, two sound data).

[0453] In this variation, the decision unit 130 decides the processing means for processing the first sound data A2 based on the first identification information A2, decides the processing means for processing the first sound data B2 based on the first identification information B2, and decides the processing means for processing the second sound data based on the second identification information.

[0454] In this variation, the first processing unit 140b processes the first sound data A2 by a processing means determined based on the first identification information A2, processes the first sound data B2 by a processing means determined based on the first identification information B2, and processes the second sound data by a processing means determined based on the second identification information.

[0455] In addition, similarly to the embodiment, the second processing means is a processing means for processing the first sound data in such a manner that the volume is attenuated by the distance D to the power of x (x≠1). Furthermore, since the first sound A2 is the first aerodynamic sound, the first identification information A2 indicates that the processing means for the first sound data A2 is the second processing means, and x is α, α is a real number, and α satisfies (Formula 5).

[0456] α>1…(Formula 5)

[0457] Furthermore, since the first sound B2 is the second aerodynamic sound, the first identification information B2 indicates that the processing means for the first sound data B2 is the second processing means, and x is β, β is a real number, and β satisfies (Formula 6).

[0458] β>2…(Formula 6)

[0459] In addition, α and β satisfy (Formula 7).

[0460] α<β…(Formula 7)

[0461] For example, α is 2 and β is 2.5.

[0462] In addition, the first sound B2 is further described. As described above, in the object information, the first sound data B2 representing the first sound B2 (second pneumatic sound) is associated with the position of the object (ambulance A). Therefore, the first processing unit 140b processes the first sound data B2 in such a way that the volume of the first sound B2 is attenuated as the distance D increases, so that the first sound B2 (second pneumatic sound) in the virtual space listened by the listener L can be made similar to the first sound B2 (second pneumatic sound) in the real space listened by the listener L.

[0463] Next, operation example 3 of the sound signal processing method performed by the sound signal processing device 100 b will be described.

[0464] [Action Example 3]

[0465] Fig.14 This is a flowchart of Operation Example 3 of the sound signal processing device 100 b according to this modification.

[0466] like Fig.14 As shown, first, the first accepting unit 160 accepts an operation of designating a value of α as an example of x represented by the first identification information A2 and an operation of designating a value of β as an example of x represented by the first identification information B2 ( S10b ). This step S10b corresponds to the first acceptance step.

[0467] Thus, x indicated by the first identification information A2 included in the object information stored in the storage unit 170 is α, and x indicated by the first identification information B2 is β.

[0468] Furthermore, the first acquisition unit 110 acquires object information including the first position information, the first audio data A2, the first identification information A2, the first audio data B2, the first identification information B2, the second audio data and the second identification information, and the second position information (S20b). This step S20b corresponds to the first acquisition step.

[0469] Next, the first calculation unit 120 calculates the distance D between the object (ambulance A) and the listener L based on the first position information included in the object information acquired by the first acquisition unit 110 and the second position information acquired by the first acquisition unit 110 ( S30 ).

[0470] Next, the decision unit 130 decides the processing means for the first audio data A2, the first audio data B2, and the second audio data based on the first identification information A2, the first identification information B2, and the second identification information included in the object information acquired by the first acquisition unit 110 (S40b). This step S40b corresponds to a decision step.

[0471] Next, the first processing unit 140b processes the first audio data A2, the first audio data B2, and the second audio data (S50b) by the processing means determined by the determination unit 130. This step S50b corresponds to the first processing step.

[0472] The first output unit 150 outputs the first audio data A2, the first audio data B2, and the second audio data processed by the first processing unit 140b (S60b). This step S60b corresponds to the first output step.

[0473] In this modification, the processing means for changing the volume of the first sound according to the first identification information and the processing means for changing the volume of the second sound according to the second identification information are provided. Therefore, the first sound and the second sound in the virtual space listened to by the listener L become sounds similar to the first sound and the second sound in the real space listened to by the listener L. Furthermore, since the first sound A2 is the first pneumatic sound and the first sound B2 is the second pneumatic sound, the second processing means is performed on the first sound data A2 and the first sound data B2 with different values ​​of x. Therefore, the first sound A2 (first pneumatic sound) and the first sound B2 (second pneumatic sound) in the virtual space listened to by the listener L become sounds similar to the first sound A2 (first pneumatic sound) and the first sound B2 (second pneumatic sound) in the real space listened to by the listener L. Based on the above, the listener L is less likely to feel awkward and the listener L can get a better sense of presence.

[0474] In addition, in the above description of this modification, the object is the ambulance A, but it is not limited to this.

[0475] In the case where the object is an object that generates multiple sounds, that is, an object associated with multiple sound data including the first sound data and the second sound data (two sound data in the above case), the same processing as in this variation is performed. The first and second examples are described below regarding the objects to which the same processing is performed.

[0476] <Example 1>

[0477] In the first example, the object is an electric fan F. Fig.15: is a diagram showing a case where the object of this modification is an electric fan F. In this case, as the first sound A2, the first aerodynamic sound, that is, the sound of wind rustling, which is generated by the movement of the electric fan F as the object, more specifically, the rotation of the blades of the electric fan F, is used. As the first sound B2, the second aerodynamic sound generated by the wind W radiated from the object (electric fan F) reaching the ear of the listener L is used as the first sound. The motor sound of the electric fan F is used as the second sound.

[0478] In the first example, the first sound and the second sound in the virtual space listened to by the listener L are also similar to the first sound and the second sound in the real space listened to by the listener L. Therefore, the listener L is less likely to feel awkward and can get a better sense of presence.

[0479] In addition, in the first example of this variation example 2, a second processing means may be used for processing the first sound data in a manner that varies the volume depending on whether the distance D is within a predetermined threshold value, as shown in the first and second examples of variation example 1 of the embodiment.

[0480] In the first and second examples of the modification example 1 of the embodiment, the second processing means is a processing means for processing the first sound data in such a manner that the volume becomes a predetermined value when the distance D is within a predetermined threshold value, but the present invention is not limited thereto. In the first example of the modification example 2, the second processing means may also be the following processing means. The second processing means of the first example of the modification example 2 may also be a processing means for processing the first sound data in such a manner that the value of x when the distance D is within the predetermined threshold value is greater than the value of x when the distance D is outside the predetermined threshold value.

[0481] <Example 2>

[0482] In the second example, the object is a zombie Z appearing in the content displayed on the display unit 30 . Fig.16 1 is a diagram showing a case where the object of the present modification is a zombie Z. The zombie Z as the object generates a first sound and a second sound. For example, the first sound is a groaning sound made by the zombie Z, and the second sound is the footsteps caused by the walking of the zombie Z. In this case, the second processing means may be a processing means for processing the first sound data in a manner that the volume is attenuated by the distance D to the power of x (x≠1), where x is a value greater than 1.

[0483] exist Fig.14 In step S50b shown, for example, the first processing unit 140b processes the first audio data by the second processing means, and processes the second audio data by the first processing means.

[0484] In the real space, the volume of a human voice (groaning sound) attenuates inversely proportionally with the increase in the distance D between the creature and the listener L. That is, if the first sound data representing the first sound as the groaning sound of the zombie Z is processed by the first processing means, the listener L hears the same sound (groaning sound) as in the real space. However, by processing the first sound data by the second processing means, that is, by processing it in a manner that the listener hears a groaning sound different from that in the real space, the listener L can experience the creepiness of the zombie Z as a fictional creature.

[0485] In the second example of this variation 2, similar to the first example of variation 2, the second processing means is a processing means for processing the first sound data in such a way that the value of x when the distance D is within a prescribed threshold value becomes a value larger than the value of x when the distance D is outside the prescribed threshold value.

[0486] (Variation 3 of the embodiment)

[0487] Hereinafter, Modification 3 of the embodiment will be described. Hereinafter, the description will be mainly focused on the differences from the embodiment, and the description of the common points will be omitted or simplified.

[0488] [constitute]

[0489] First, the configurations of the information generating device 40 and the sound signal processing device 200 according to Modification 3 of the present embodiment will be described. Fig.17 1 is a block diagram showing the functional configuration of the information generating device 40 and the sound signal processing device 200 according to the present modification.

[0490] The information generating device 40 and the acoustic signal processing device 200 according to the present modification can suppress the occurrence of the problem described in (the knowledge underlying the present disclosure) that the listener L hears the second aerodynamic sound which makes him feel uncomfortable. Fig.18 1 is a diagram showing an electric fan F and a listener L as objects according to the present modification example.

[0491] The information generating device 40 generates first target audio information and outputs it to the sound signal processing device 200. The sound signal processing device 200 acquires the output first target audio information and outputs sound data to the headphone 20 based on the acquired first target audio information.

[0492] First of all, Fig.17 The information generating device 40 shown will be described.

[0493] The information generating device 40 includes a receiving unit 41 , a second acquiring unit 42 , a first generating unit 43 , an output unit 44 , and a storage unit 45 .

[0494] The receiving unit 41 receives an operation from a user (for example, a creator of content executed in a virtual space) of the information generating device 40. Specifically, the receiving unit 41 is implemented by hardware buttons, but may be implemented by a touch panel or the like.

[0495] The second acquisition unit 42 acquires first sound data indicating a first sound generated at a position associated with the position of the listener L in the virtual space and first position information indicating the position of an object in the virtual space. In this modification, the object is an object radiating wind W, such as Fig.18 The electric fan F is shown. In addition, in the virtual space, the listener L is in a position where he is bathed in the wind W emitted from the electric fan F. The first sound is a sound generated at a position associated with the position of the listener L, and here, the position associated with the position of the listener L is the position of the ear of the listener L. That is, the first sound is a sound generated at a position associated with the position of the listener L (that is, the position of the ear of the listener L) due to an object. More specifically, the first sound related to this modification is an aerodynamic sound (second aerodynamic sound) generated by the wind W emitted from the electric fan F as an object reaching the ear of the listener L.

[0496] In this modification, the receiving unit 41 receives an operation indicating the first sound data and the first position information from the user. That is, when the user operates the receiving unit 41 to input the first sound data and the first position information, the second acquiring unit 42 acquires the input first sound data and the first position information.

[0497] The first generating unit 43 generates first object audio information including information related to the object for reproducing the first sound at a position associated with the position of the listener L due to the object and the first position information based on the first sound data and the first position information acquired by the second acquiring unit 42 .

[0498] The information on the object indicates that the object is the electric fan F and that the object reproduces the first sound by itself.

[0499] Furthermore, in the first object audio information related to the present modification, the first sound data indicating the first sound (second aerodynamic sound) generated by the object at the position associated with the position of the listener L is associated with the position of the object (electric fan F) indicated by the first position information. Furthermore, in the real space, the second aerodynamic sound is generated at the ear of the listener L, but here the position of the electric fan F in the virtual space is regarded as the position of the sound source.

[0500] In addition, the first generating unit 43 may generate the first target audio information including the directivity information and the unit distance information.

[0501] Directivity information is information indicating characteristics according to the direction of wind W radiated from electric fan F. Directivity information is a database storing wind speeds for each direction in which wind W is radiated or a decay rate of the volume of the second aerodynamic sound in the SOFA (Spatially Oriented Format for Acoustics) format, for example.

[0502] The unit distance information is information including a unit distance as a reference distance and aerodynamic sound data indicating aerodynamic sound at a position away from the position of the object indicated by the first position information by a unit distance. The aerodynamic sound data indicated by the unit distance information is data indicating aerodynamic sound (second aerodynamic sound) at a position away from the position of the object by a unit distance in the front direction of the wind W radiated by the object indicated by the directivity information.

[0503] Here, use Fig.19 Describes the directivity information and unit distance information.

[0504] Fig.19 This is a diagram for explaining the directional information and unit distance information of this modification. Let the front direction of the wind W emitted from the object (electric fan F) be direction Df. Here, let the direction that forms an angle of θ1 with direction Df be direction D31, and let the direction that forms an angle of θ2 with direction Df be direction D32. Fig.19 The unit distance is shown in , and a circle is shown at a position away from the object (electric fan F) by the unit distance. In other words, the radius of the circle is the unit distance.

[0505] Let the wind speed of the wind W at a position that is a unit distance away from the object (electric fan F) in the direction DF be wsF. Also, if the wind speed of the wind W at a position that is a unit distance away from the object (electric fan F) in the direction D31 is ws1, and the wind speed of the wind W at a position that is a unit distance away from the object (electric fan F) in the direction D32 is ws2, then ws1 and ws2 satisfy (Formula 8) and (Formula 9).

[0506] ws1=wsF×C1…(Formula 8)

[0507] ws2=wsF×C2…(Formula 9)

[0508] Directivity information is a database showing values ​​of C1 and C2 (C1 and C2 are constants) for each angle such as θ1 and θ2. C1 is recorded as the value at angle θ1 indicated by the directivity information, and C2 is recorded as the value at angle θ2 indicated by the directivity information.

[0509] In the above example, the directivity related to the wind speed is described, but the directivity related to the volume of the sound (second aerodynamic sound) caused by the wind W can also be described in the same way.

[0510] Furthermore, the first generating unit 43 generates the first object audio information further including flag information. The flag information is information indicating whether to perform a process of convolving the sound data (here, aerodynamic sound data) representing the sound generated from the object (electric fan F) with a head-related transfer function depending on the direction in which the sound arrives when reproducing the sound.

[0511] Directivity information, unit distance information, and mark information may be pre-stored in, for example, storage unit 45. First generator 43 may obtain directivity information, unit distance information, and mark information from storage unit 45, or may obtain the information through operation receiver 41 in the same manner as first sound data and first position information.

[0512] The output unit 44 outputs the first sound data and the first position information acquired by the second acquisition unit 42 and the first target audio information generated by the first generation unit 43 to the acoustic signal processing device 200 .

[0513] The storage unit 45 is a storage device that stores computer programs and the like executed by the receiving unit 41 , the second acquiring unit 42 , the first generating unit 43 , and the output unit 44 .

[0514] Furthermore, Fig.17 The audio signal processing device 200 shown will be described.

[0515] like Fig.17 As shown, the sound signal processing device 200 includes a third acquisition unit 210 , a second calculation unit 220 , a second processing unit 240 , a second output unit 250 , and a storage unit 270 .

[0516] The third acquisition unit 210 acquires the first target audio information generated by the information generation device 40, the first sound data acquired by the information generation device 40, and the second position information indicating the position of the listener L of the first sound. The third acquisition unit 210 acquires the second position information from the earphone 20 (more specifically, from the head sensor unit 21), but is not limited thereto.

[0517] The second calculation unit 220 calculates the distance D between the object (the electric fan F) and the listener L based on the first position information included in the first target audio information obtained by the third acquisition unit 210 and the obtained second position information. The second calculation unit 220 calculates the distance D by the same method as the first calculation unit 120 of the related embodiment.

[0518] Furthermore, the second calculation unit 220 calculates the direction between two points connecting the object (the electric fan F) and the listener L based on the first position information included in the first target audio information acquired by the third acquisition unit 210 and the acquired second position information.

[0519] The second processing unit 240 processes the first sound data in such a manner that the volume of the first sound decreases as the distance D calculated by the second calculating unit 220 increases. For example, the second processing unit 240 may process the first sound data by the second processing means described in the second modification of the embodiment. As described above, since the first sound is the second aerodynamic sound, the first sound data may be processed using β as the value of x. In this case, β is 2.5, for example.

[0520] In addition, in the first object audio information related to the present variation, the first sound data representing the first sound (second pneumatic sound) is associated with the position of the object (electric fan F). Therefore, the second processing unit 240 processes the first sound data B2 in such a manner that the volume of the first sound B2 decreases as the distance D increases, thereby making the first sound (second pneumatic sound) in the virtual space listened to by the listener L similar to the first sound (second pneumatic sound) in the real space listened to by the listener L.

[0521] Furthermore, when the calculated distance D is greater than the unit distance indicated by the unit distance information included in the first object audio information acquired by the third acquisition unit 210, the second processing unit 240 may perform the following processing. That is, in this case, the second processing unit 240 processes the first sound data in such a manner that the volume of the first sound is attenuated according to the calculated distance D and the unit distance. Fig. 20 This processing will be described.

[0522] Fig. 20 2 is a diagram for explaining the processing of the second processing unit 240 according to this modification. Fig. 20 The positional relationship between the object (electric fan F) and the listener L is shown in FIG. The listener L is located in the front direction of the wind W emitted from the object (electric fan F). As described above, the unit distance information shows the aerodynamic sound data representing the aerodynamic sound at a position away from the position of the object by a unit distance in the front direction. Fig. 20 In the example shown, the first sound data corresponds to the pneumatic sound data. When the calculated distance D is greater than the unit distance, the second processing unit 240 processes the pneumatic sound data representing the pneumatic sound at a position away from the position of the object by the unit distance so that the volume of the first sound (second pneumatic sound) is attenuated as the distance D increases.

[0523] Furthermore, the second processing unit 240 may process the first sound data in such a manner as to control the volume of the first sound based on the characteristics represented by the angle between the front direction of the wind W emitted from the object (electric fan F) and the direction between the two points calculated by the second calculation unit 220 and the directivity information. Fig.21 This processing will be described.

[0524] Fig.21 2 is a diagram for explaining other processing of the second processing unit 240 in this modification. Fig.21 The positional relationship between the object (electric fan F) and the listener L is shown in FIG. The listener L is located in the direction D31 relative to the object (electric fan F). In this case, the angle between the front direction and the direction between the two points is θ1. Compared with the case where the listener L is located in the front direction (the position of the listener L indicated by the dotted line), the angle between the two points is θ1. Fig.21 In the case of the position of the listener L shown (the position of the listener L indicated by the solid line), the wind speed of the wind W to which the listener L is exposed becomes lower.

[0525] In Fig.21 The wind speed received by the listener L at the position shown is a value obtained by multiplying the wind speed received by the listener L at the front position by the above C1. Since the volume of the first sound (second aerodynamic sound) heard by the listener L changes according to the wind speed received by the listener L, the second processing unit 240 can process the first sound data (here, the aerodynamic sound data represented by the unit distance information) according to the wind speed.

[0526] In addition, the second processing unit 240 can perform Fig. 20 The processing described in Fig.21 Both of the processes described in the above description output a first sound data (pneumatic sound data) after the processing to the second output unit 250.

[0527] The second output unit 250 outputs the first sound data (pneumatic sound data) processed by the second processing unit 240. Here, the second output unit 250 outputs the first sound data to the earphone 20. Thus, the earphone 20 can reproduce the first sound represented by the output first sound data.

[0528] The storage unit 270 is a storage device that stores computer programs and the like executed by the third acquisition unit 210 , the second calculation unit 220 , the second processing unit 240 , and the second output unit 250 .

[0529] Hereinafter, operation example 4 of the information generating method performed by the information generating device 40 and operation example 5 of the sound signal processing method performed by the sound signal processing device 200 will be described.

[0530] [Action Example 4]

[0531] Fig. 22 This is a flowchart of operation example 4 of the information generating device 40 according to this modification.

[0532] like Fig. 22 As shown, first, the receiving unit 41 receives an operation indicating the first audio data and the first position information from the user (S110). That is, the first audio data and the first position information are input by the user operating the receiving unit 41.

[0533] Next, the second acquisition unit 42 acquires first sound data representing a first sound generated at a position associated with the position of the listener L in the virtual space and first position information representing the position of an object in the virtual space (S120). Here, the second acquisition unit 42 acquires the first sound data and the first position information input in step S110. This step S120 corresponds to the second acquisition step.

[0534] Next, the first generating unit 43 generates the first object audio information including the information related to the object, the first position information, the unit distance information and the directivity information based on the first sound data and the first position information acquired by the second acquiring unit 42 (S130). In addition, the generated first object audio information may also include the flag information. This step S130 is equivalent to the first generating step.

[0535] Furthermore, the output unit 44 outputs the first sound data and the first position information acquired by the second acquisition unit 42 and the first target audio information generated by the first generation unit 43 to the acoustic signal processing device 200 ( S140 ).

[0536] [Action Example 5]

[0537] Fig.23 This is a flowchart of Operation Example 5 of the sound signal processing device 200 according to this modification.

[0538] like Fig.23 As shown, first, the third acquisition unit 210 acquires the first target audio information generated by the information generating device 40, the first sound data acquired by the information generating device 40, and the second position information output from the headphone 20 (S210). This step S210 corresponds to the third acquisition step.

[0539] Next, the second calculation unit 220 calculates the distance D between the object (electric fan F) and the listener L based on the first position information included in the first object audio information obtained by the third acquisition unit 210 and the obtained second position information, and calculates the direction between two points connecting the object (electric fan F) and the listener L based on the first position information included in the first object audio information obtained by the third acquisition unit 210 and the obtained second position information (S220). This step S220 corresponds to the second calculation step.

[0540] Next, the second processing unit 240 processes the first sound data (S230) in a manner that controls the volume of the first sound based on the characteristics represented by the angle between the above-mentioned front direction and the direction between the two points calculated by the second calculation unit 220 and the directivity information, and in a manner that attenuates the volume of the first sound according to the calculated distance D and the unit distance when the calculated distance D is larger than the unit distance represented by the unit distance information included in the first object audio information obtained by the third acquisition unit 210.

[0541] Furthermore, when the first target audio information includes flag information, the second processing unit 240 determines whether to perform a head-related transfer function process that depends on the direction of sound arrival on the first sound signal based on the first sound data according to the flag information. Here, as an example, the second processing unit 240 performs a head-related transfer function process that depends on the direction of sound arrival on the first sound signal based on the first sound data according to the flag information (S240). Steps S230 and S240 correspond to the second processing step.

[0542] Next, the second output unit 250 outputs the first audio data (first audio signal) processed by the second processing unit 240 (S250). This step S250 corresponds to the second output step.

[0543] The information generation method related to this variation can generate the first object audio information in which the first sound data representing the first sound (second pneumatic sound) generated at the position associated with the position of the listener L due to the object is associated with the position of the object. Furthermore, the sound signal processing method related to this variation processes the first sound data in such a way that the volume of the first sound (second pneumatic sound) is attenuated as the distance D between the object and the listener L increases, so that the first sound (second pneumatic sound) in the virtual space listened to by the listener L becomes a sound similar to the first sound (second pneumatic sound) in the real space listened to by the listener L. That is, the listener L does not hear the first sound (second pneumatic sound) that feels awkward, and the listener L can obtain a sense of presence.

[0544] (Variation 4 of the embodiment)

[0545] Hereinafter, Modification 4 of the embodiment will be described. Hereinafter, the differences from Modification 3 of the embodiment will be mainly described, and the description of the common points will be omitted or simplified.

[0546] [constitute]

[0547] First, the configurations of the information generating device 40 and the sound signal processing device 200 c according to Modification 4 of the present embodiment will be described. Fig.24 1 is a block diagram showing the functional configuration of the information generating device 40 and the sound signal processing device 200 c according to the present modification.

[0548] In this modification, information generating device 40 according to modification 3 is used. Acoustic signal processing device 200c has the same configuration as acoustic signal processing device 200 according to modification 3 except that second calculation unit 220 is not included and second processing unit 240c is included instead of second processing unit 240.

[0549] In this modification, the sound signal processing device 200c is a device that processes a first sound generated by an object at a position associated with the position of the listener L and a second sound caused by the object. The object in this modification is the ambulance A as in the second modification.

[0550] The first sound is a pneumatic sound generated by the wind W radiated from the object (ambulance A) reaching the ear of the listener L, that is, the second pneumatic sound. The second sound is the whistle sound emitted from the ambulance A.

[0551] like Fig.24 As shown, the sound signal processing device 200 c includes a third acquisition unit 210 , a second processing unit 240 c , a second output unit 250 , and a storage unit 270 .

[0552] The third acquisition unit 210 related to this variation acquires the first object audio information generated by the information generating device 40, the first sound data acquired by the information generating device 40, and the second object audio information. The second object audio information is information that associates the first position information indicating the position of the object (ambulance A) in the virtual space with the second sound data indicating the second sound caused by the object (ambulance A). In addition, since the second object audio information is data that associates the second sound data indicating the second sound with the first position information indicating the position where the second sound is generated, it is equivalent to the object audio information in the prior art such as the technology disclosed in Patent Document 1.

[0553] The second target audio information may be generated by the information generating device 40 and output to the sound signal processing device 200c. The third acquisition unit 210 acquires the output second target audio information.

[0554] The second processing unit 240 c processes the first audio data acquired by the third acquisition unit 210 and the second audio data included in the second target audio information acquired by the third acquisition unit 210 as follows.

[0555] First, the processing on the second audio data will be described.

[0556] The second processing unit 240 c performs a process of convolving a second audio signal based on the second audio data represented by the second target audio information acquired by the third acquisition unit 210 with a head-related transfer function that depends on the direction in which the sound arrives.

[0557] Next, the processing of the first audio data will be described.

[0558] As an example, the second processing unit 240c does not perform a process of convolving the first sound signal based on the first sound data acquired by the third acquisition unit 210 with a head-related transfer function that depends on the direction from which the sound arrives. In addition, as another example, the second processing unit 240c performs a process of convolving the first sound signal based on the first sound data acquired by the third acquisition unit 210 with a process that depends on the direction from which the wind W arrives from the ambulance A to the listener L. Fig.25 The processing of this other example will be described.

[0559] Fig.25 This is a diagram for explaining the processing on the first audio data according to the present modification example.

[0560] like Fig.25 As shown, depending on the positional relationship between the object (ambulance A) and the listener L, the speed and volume of wind W reaching the right ear and left ear of the listener L are different. Fig.25 In the figure, at the left ear, the wind W arrives faster and the wind volume is larger, and at the right ear, the wind W arrives slower and the wind volume is smaller.

[0561] Therefore, the second processing unit 240c can perform the following processing on the first sound data as a process depending on the direction of the wind W coming from the ambulance A to the listener L. That is, the first sound data can be processed in such a way that the first sound (second aerodynamic sound) heard by the listener L becomes a sound that simulates the time difference and wind volume ratio between the two ears of the listener L. As a result, the listener L can perceive the direction of the source of the wind W.

[0562] In addition, the first object audio information includes processing information indicating whether the second processing unit 240c performs no convolution head correlation transfer function processing on the first sound signal or performs processing on the first sound signal depending on the direction of the wind W coming from the ambulance A to the listener L. The second processing unit 240c processes the first sound signal according to the processing information included in the first object audio information.

[0563] The second output unit 250 of this modification outputs the second sound signal processed by the second processing unit 240c. When the second processing unit 240c does not process the first sound signal, the second output unit 250 outputs the unprocessed first sound signal. In addition, when the second processing unit 240c processes the first sound signal, the second output unit 250 outputs the processed first sound signal.

[0564] Furthermore, the storage unit 270 according to the present modification stores the head-related transfer function used by the second processing unit 240c and information necessary for processing depending on the direction in which the wind W comes from the ambulance A to the listener L.

[0565] Hereinafter, operation examples 5 and 6 of the sound signal processing method performed by the sound signal processing device 200 c will be described.

[0566] [Action Example 6]

[0567] Fig.26 2 is a flowchart of Operation Example 6 of the sound signal processing device 200c according to this modification. In this operation example, the second processing unit 240c does not process the first sound signal.

[0568] like Fig.26 As shown, first, the third acquisition unit 210 acquires the first object audio information generated by the information generating device 40, the first sound data acquired by the information generating device 40, and the second object audio information in which the first position information and the second sound data are associated (S310). This step S310 is equivalent to the third acquisition step.

[0569] The second processing unit 240c does not perform the process of convolving the first audio signal based on the first audio data acquired by the third acquisition unit 210 with the head-related transfer function that depends on the direction in which the sound arrives (S320).

[0570] The second processing unit 240c performs a process (S330) of convolving a second audio signal based on the second audio data represented by the second target audio information acquired by the third acquisition unit 210 with a head-related transfer function that depends on the direction in which the sound arrives. These steps S320 and S330 correspond to the second processing step.

[0571] The second output unit 250 outputs the first audio signal that has not been processed by the second processing unit 240c and the second audio signal that has been processed by the second processing unit 240c (S340). This step S340 corresponds to the second output step.

[0572] [Action Example 7]

[0573] Fig. 272 is a flowchart of Operation Example 7 of the sound signal processing device 200c according to this modification. In this operation example, the second processing unit 240c is an example of processing the first sound signal.

[0574] like Fig. 27 As shown, first, step S310 is performed.

[0575] The second processing unit 240c performs processing depending on the direction of the wind W coming from the ambulance A toward the listener L on the first sound signal based on the first sound data acquired by the third acquisition unit 210 (S320c).

[0576] Next, proceed to step S330.

[0577] Next, the second output unit 250 outputs the first audio signal processed by the second processing unit 240c and the second audio signal processed by the second processing unit 240c (S340c).

[0578] In this modification, since the head correlation transfer function is convolved on the second sound signal based on the second sound data, the second sound in the virtual space listened to by the listener L becomes a sound similar to the second sound in the real space listened to by the listener L. Therefore, the listener L is less likely to feel awkward, and the listener L can obtain a better sense of presence.

[0579] Furthermore, in Action Example 7, since the first sound signal based on the first sound data is processed depending on the direction of the wind W, the first sound in the virtual space listened to by the listener L becomes a sound similar to the first sound in the real space listened to by the listener L. Therefore, the listener L is less likely to feel awkward, and the listener L can obtain a better sense of presence.

[0580] (Variation 5 of the embodiment)

[0581] Hereinafter, Modification 5 of the embodiment will be described. Hereinafter, the differences from Modification 4 of the embodiment will be mainly described, and the description of the common points will be omitted or simplified.

[0582] [constitute]

[0583] First, the configurations of the information generating device 40 and the sound signal processing device 200d according to Modification 5 of the present embodiment will be described. Fig.28 2 is a block diagram showing the functional configuration of the information generating device 40 and the sound signal processing device 200d according to the present modification.

[0584] In this modification, information generating device 40 according to modification 4 is used. Sound signal processing device 200d has the same configuration as sound signal processing device 200c according to modification 4, except that a second processing unit 240d is provided instead of second processing unit 240c.

[0585] In Modification 4, the first sound (second aerodynamic sound) generated at a position associated with the position of the listener L due to the ambulance A as one object and the second sound (whistle sound) caused by the ambulance A as one object are handled. In Modification 5, the first sound (second aerodynamic sound) generated at a position associated with the position of the listener L due to the ambulance A as one object and the third sound caused by another object in the virtual space that is different from the one object are handled.

[0586] The other object is similar to the first object, and is not particularly limited as long as it is an object included in the content displayed by the display unit 30 that displays the content executed in the virtual space (here, as an example, a video). The first object and the other object are set in the same virtual space. In addition, for simplicity, the first object is sometimes simply referred to as an object, and the other object is simply referred to as another object.

[0587] The third sound is a sound generated at the position of another object in the virtual space. In addition, the third sound is a sound different from the first and second pneumatic sounds.

[0588] In this way, in this modification, the first sound and the third sound are handled, and more specifically, a plurality of objects including one object and another object are handled.

[0589] The third acquisition unit 210 of this variant acquires the first object audio information generated by the information generating device 40, the first sound data acquired by the information generating device 40, and the third object audio information. The third object audio information is information that associates the third position information indicating the position of other objects in the virtual space with the third sound data indicating the third sound generated at the position of the other objects. In addition, since the third object audio information is data that associates the third sound data indicating the third sound with the third position information indicating the position where the third sound is generated, it is equivalent to the object audio information in the prior art such as the technology disclosed in Patent Document 1.

[0590] The third target audio information may be generated by the information generating device 40 and output to the sound signal processing device 200d. The third acquisition unit 210 acquires the output third target audio information.

[0591] The second processing unit 240 d processes the first audio data acquired by the third acquisition unit 210 and the third audio data included in the third target audio information acquired by the third acquisition unit 210 as follows.

[0592] The second processing unit 240d processes the first sound signal based on the first sound data in a direction dependent on the wind W coming from the ambulance A to the listener L. In addition, the second processing unit 240d performs convolution processing on the third sound signal based on the third sound data represented by the acquired third object audio information, in a head-related transfer function-dependent direction of the sound arrival. That is, in this modified example, the second processing unit 240d performs convolution processing on the first sound signal with the head-related transfer function-dependent direction of the sound arrival. Fig.25 The second processing unit 240d performs the same processing as that of the second audio signal in Modification Example 4 on the third audio signal.

[0593] Furthermore, the second output unit 250 according to the present modification outputs the first audio signal processed by the second processing unit 240d and the third audio signal processed by the second processing unit 240d.

[0594] Next, operation example 8 of the sound signal processing method performed by the sound signal processing device 200d will be described.

[0595] [Action Example 8]

[0596] Fig.29 This is a flowchart of Operation Example 8 of the sound signal processing device 200d according to this variation.

[0597] like Fig.29 As shown, first, the third acquisition unit 210 acquires the first object audio information generated by the information generating device 40, the first sound data acquired by the information generating device 40, and the third object audio information that associates the third position information indicating the position of the other object with the third sound data indicating the third sound generated at the position of the other object (S310d). This step S310d is equivalent to the third acquisition step.

[0598] The second processing unit 240d performs processing depending on the direction of the wind W coming from the ambulance A toward the listener L on the first sound signal based on the first sound data acquired by the third acquisition unit 210 (S320c).

[0599] The second processing unit 240d performs convolution processing (S330d) on the third audio signal based on the third audio data represented by the third target audio information acquired by the third acquisition unit 210 with the head-related transfer function depending on the direction of arrival of the sound. These steps S320c and S330d correspond to the second processing step.

[0600] Next, the second output unit 250 outputs the first audio signal processed by the second processing unit 240d and the third audio signal processed by the second processing unit 240d (S340d). This step S340d corresponds to the second output step.

[0601] In this modification, when a plurality of objects including an object and other objects are provided in the virtual space, the first sound and the third sound in the virtual space listened to by the listener L become sounds similar to the first sound and the third sound in the real space listened to by the listener L. Therefore, the listener L is less likely to feel awkward, and the listener L can obtain a better sense of presence.

[0602] (Variation 6 of the embodiment)

[0603] Hereinafter, Modification 6 of the embodiment will be described. Hereinafter, the differences from Modification 3 of the embodiment will be mainly described, and the description of the common points will be omitted or simplified.

[0604] [constitute]

[0605] First, the configurations of information generating device 50 and sound signal processing device 400 according to Modification 6 of the present embodiment will be described. Fig.30 1 is a block diagram showing the functional configuration of information generating device 50 and sound signal processing device 400 according to this modification.

[0606] The information generating device 50 and the sound signal processing device 400 of this modification example are similar to those of the modification example 3, and can suppress the occurrence of the problem of the second aerodynamic sound that the listener L hears and feels uncomfortable, as described in (the knowledge that is the basis of this disclosure). In this modification example, as in the modification example 3, the object is an electric fan F, but it is not limited to this, as long as it can emit wind W. In addition, the wind W emitted from the object is as follows: Fig.13 As shown, the wind generated by the movement of the target (ambulance A) such as a moving body is also included.

[0607] The information generating device 50 generates the fourth target audio information and outputs it to the sound signal processing device 400. The sound signal processing device 400 acquires the output fourth target audio information and outputs the sound data to the headphone 20 based on the acquired fourth target audio information.

[0608] First of all, Fig.30 The information generating device 50 shown will be described.

[0609] The information generating device 50 includes a second receiving unit 51 , a fourth acquiring unit 52 , a second generating unit 53 , a third output unit 54 , and a storage unit 55 .

[0610] The second accepting unit 51 accepts an operation from a user (for example, a creator of content executed in a virtual space) of the information generating device 50. Specifically, the second accepting unit 51 is implemented by a hardware button, but may be implemented by a touch panel or the like.

[0611] The fourth acquisition unit 52 acquires a generation position of a first wind blowing in the virtual space, a first wind direction of the first wind, and a first assumed wind speed.

[0612] In this modification, the first wind blowing in the virtual space is the wind W radiated from the electric fan F as the object. That is, the location where the first wind is generated is the location where the electric fan F is configured. In addition, in this modification, as long as the first wind blows in the virtual space, the electric fan F as the object radiating the first wind does not need to be configured in the virtual space (more specifically, the virtual space where the listener L is located). That is, the electric fan F is configured outside the virtual space where the listener L is located, and the first wind as the wind W radiated from the electric fan F reaches the virtual space. In addition, of course, the electric fan F as the object radiating the first wind may also be configured in the virtual space where the listener L is located.

[0613] The first wind direction is the direction of the first wind, which is the front direction of the wind W radiated from the object (electric fan F). Fig.19 The first assumed wind speed may be the wind speed of the first wind, which is the wind speed of the first wind at a position that is a unit distance away from the generating position as a reference in the direction of the first wind direction. That is, the first assumed wind speed is, for example, Fig.19 The wind speed wsF is shown.

[0614] As described above, since the wind W radiated from the target fan F, namely the first wind, blows in the virtual space, the listener L hears the aerodynamic sound (second aerodynamic sound) generated by the wind W (first wind) reaching the listener L's ears.

[0615] In addition, in this modified example, the second receiving unit 51 receives an operation indicating the generation position of the first wind, the first wind direction, and the first assumed wind speed from the user. That is, the user operates the second receiving unit 51 to input the generation position of the first wind, the first wind direction, and the first assumed wind speed, and the fourth obtaining unit 52 obtains the input generation position of the first wind, the first wind direction, and the first assumed wind speed.

[0616] The second accepting unit 51 preferably accepts an operation from the user to designate that the unit distance is the second designated value. That is, the second accepting unit 51 is operated by the user to set the unit distance to the second designated value.

[0617] The second generating unit 53 generates fourth target audio information in which the generation position, the first wind direction, and the first assumed wind speed acquired by the fourth acquiring unit 52 are associated with each other.

[0618] In addition, the second receiving unit 51 may receive an operation from the user to designate directional information indicating characteristics corresponding to the direction of the first wind. Fig.19 When such an operation is accepted, the second generating unit 53 generates fourth target audio information in which the generation position, the first wind direction, and the first assumed wind speed are associated with the directional information indicated by the operation accepted by the second accepting unit 51 .

[0619] The storage unit 55 is a storage device that stores computer programs and the like executed by the second receiving unit 51, the fourth acquiring unit 52, the second generating unit 53, and the third output unit 54. In addition, it is assumed that the storage unit 55 stores aerodynamic sound core information including representative wind speed and aerodynamic sound data representing aerodynamic sound (second aerodynamic sound) generated by wind representing the wind speed reaching the ears of the listener L in the virtual space. However, the aerodynamic sound core information does not necessarily have to be stored in the information generating device 50, and may be stored in a memory independent of the information generating device 50.

[0620] In this modification, the second accepting unit 51 accepts an operation indicating the pneumatic sound core information from the user. That is, the user operates the second accepting unit 51 to input the pneumatic sound core information, and the input pneumatic sound core information is stored in the storage unit 55.

[0621] The pneumatic sound core information shows a representative wind speed as an example of a value of one wind speed, and pneumatic sound data indicating pneumatic sound generated by wind reaching the ears of the listener L at the representative wind speed, which is a database used in information processing in the acoustic signal processing device 400 described later. The pneumatic sound data indicating the pneumatic sound (the second pneumatic sound) indicates the volume of the second pneumatic sound, etc. In addition, the pneumatic sound core information may include data indicating the distribution of frequency components of the pneumatic sound (hereinafter sometimes referred to as frequency data). For example, the frequency data is data indicating the frequency characteristics of the pneumatic sound (the second pneumatic sound).

[0622] Such aerodynamic sound core information can also be used in the third processing unit 440 described later to obtain the volume of the second aerodynamic sound heard by the listener L based on the wind speed reaching the ear.

[0623] In addition, the aerodynamic sound core information may also include a plurality of sets of representative wind speeds and aerodynamic sound data at the representative wind speeds. Figure 4CAs shown in FIG. 1 , because the distance from the source of the wind W is different, the wind speed at each position with different wind speeds can be set as a representative wind speed, and a group consisting of the representative wind speed and the frequency data corresponding to the representative wind speed is formed, and the aerodynamic sound core information includes a plurality of such groups. The aerodynamic sound core information thus formed can also be used in the third processing unit 440 described later to obtain the volume of the second aerodynamic sound heard by the listener L using the wind speed reaching the ear as an index.

[0624] The third output unit 54 outputs the fourth target audio information generated by the second generation unit 53 and the aerodynamic sound core information stored in the storage unit 55 to the acoustic signal processing device 400 .

[0625] Furthermore, Fig.30 The audio signal processing device 400 shown will be described.

[0626] like Fig.30 As shown, the sound signal processing device 400 includes a fifth acquisition unit 410 , a third calculation unit 420 , a third processing unit 440 , a fourth output unit 450 , and a storage unit 470 .

[0627] The fifth acquisition unit 410 acquires the fourth target audio information and the aerodynamic sound core information output by the information generating device 50, and the second position information indicating the position of the listener L in the virtual space. The fifth acquisition unit 410 acquires the second position information from the earphone 20 (more specifically, the head sensor unit 21), but the present invention is not limited thereto. In addition, the aerodynamic sound core information here includes frequency data.

[0628] The third calculation unit 420 calculates the distance D between the generation position (i.e., the fan F) and the listener L based on the generation position included in the fourth target audio information acquired by the fifth acquisition unit 410 and the acquired second position information. The third calculation unit 420 calculates the distance D by the same method as the first calculation unit 120 of the related embodiment.

[0629] Furthermore, the third calculation unit 420 calculates the direction between two points connecting the object (the electric fan F) and the listener L based on the generation position included in the fourth target audio information acquired by the fifth acquisition unit 410 and the acquired second position information.

[0630] The third processing unit 440 processes the pneumatic sound data so that the volume of the pneumatic sound decreases as the distance D calculated by the third calculating unit 420 increases.

[0631] Here, the third processing unit 440 can process the aerodynamic sound data in a manner in which the volume of the aerodynamic sound (the second aerodynamic sound) is attenuated to the γth power of a value obtained by dividing the representative wind speed by the wind speed at the ear, which is the wind speed of the first wind when reaching the ear of the listener L. That is, the third processing unit 440 processes the aerodynamic sound data based on the distance D, the representative wind speed, and the wind speed at the ear. In addition, for the wind speed at the ear, the wind speed slows down as the distance D calculated by the third calculation unit 420 increases. In addition, the wind speed at the ear is a value attenuated to the zth power of a value obtained by dividing the distance D calculated by the third calculation unit 420 by the unit distance.

[0632] More specifically, the third processing unit 440 performs the following processing.

[0633] First, assume that the first wind direction (for example Fig.19 The angle formed by the direction Df shown in FIG. 1 and the calculated direction between the two points is θ. In this case, if the wind speed of the first wind reaching the ear of the listener L is Se1, Se1 satisfies (Formula 10).

[0634] Se1 = first assumed wind speed × value at angle θ indicated by directivity information × {(unit distance as a reference distance / distance D)}^z ... (Formula 10)

[0635] For example, if θ is θ1, the value at the angle θ (angle θ1) indicated by the directivity information is C1. In addition, let the value obtained by dividing the representative wind speed by the wind speed reaching the ear Se1 be R1. Furthermore, V3, which is the volume of the second aerodynamic sound heard by the listener L, satisfies (Formula 11).

[0636] V3 = volume represented by the pneumatic sound data of the pneumatic sound core information × {(1 / R1)}^γ ... (Formula 11)

[0637] Furthermore, z preferably satisfies (Formula 12).

[0638] z=1…(Formula 12)

[0639] Furthermore, γ and z preferably satisfy (Formula 13).

[0640] γ×z<4…(Formula 13)

[0641] The method of calculating the volume V3 of the second aerodynamic sound heard by the listener L based on the wind speed Se1 reaching the ear is not limited to the above. For example, the aerodynamic sound core information may include a plurality of groups of representative wind speeds and frequency data at the wind speeds, and the third processing unit 440 detects the frequency data corresponding to the representative wind speed close to Se1, and generates the second aerodynamic sound with a specified volume by applying the frequency data. That is, instead of calculating the volume V3 by mathematical calculation based on Se1, the aerodynamic sound data may be detected with Se1 as an index and the specified volume may be achieved by applying the aerodynamic sound data. Since the volume of the second aerodynamic sound is smaller as the wind speed is smaller, the aerodynamic sound core information should be provided so that if the frequency data representing a smaller wind speed is applied, the resulting volume becomes smaller. Since Se1 becomes smaller as the distance D increases, it is possible to control so that the sound V3 becomes smaller as the distance D increases.

[0642] The fourth output unit 450 outputs the pneumatic sound data processed by the third processing unit 440. Here, the fourth output unit 450 outputs the pneumatic sound data to the earphone 20. Thus, the earphone 20 can reproduce the second pneumatic sound represented by the output pneumatic sound data.

[0643] The storage unit 470 is a storage device that stores computer programs and the like executed by the fifth acquisition unit 410 , the third calculation unit 420 , the third processing unit 440 , and the fourth output unit 450 .

[0644] Furthermore, when the aerodynamic sound core information acquired by the fifth acquisition unit 410 includes frequency data, the third processing unit 440 may perform the following processing.

[0645] The third processing unit 440 processes the pneumatic sound data so that the distribution of the frequency component of the pneumatic sound (second pneumatic sound) is shifted toward the lower frequency side as the distance D calculated by the third calculating unit 420 increases.

[0646] Here, the third processing unit 440 processes the aerodynamic sound data in such a manner that the distribution of the frequency components of the aerodynamic sound is shifted to a frequency that is a reciprocal multiple of a value obtained by dividing the representative wind speed by the wind speed reaching the ear (the above-mentioned R1). That is, the distribution of the frequency components included in the aerodynamic sound core information (the distribution of the frequency components before processing) is processed by the third processing unit 440 to become a distribution of the frequency components after being shifted to a frequency obtained by multiplying the reciprocal of R1 by the frequency.

[0647] In this way, the third processing unit 440 processes the aerodynamic sound data based on the distance D, the representative wind speed, and the wind speed at the ear. Regarding the wind speed at the ear, the more the calculated distance D increases, the slower the wind speed. In addition, the wind speed at the ear is a value attenuated by the z-th power of the value obtained by dividing the calculated distance D by the unit distance. In addition, in this case, z also satisfies the above (Formula 12).

[0648] Hereinafter, operation example 9 of the information generating method performed by the information generating device 50 and operation examples 10 and 11 of the sound signal processing method performed by the sound signal processing device 400 will be described.

[0649] [Action Example 9]

[0650] Fig.31 This is a flowchart of Operation Example 9 of the information generating device 50 according to this modification.

[0651] like Fig.31 As shown, first, the second accepting unit 51 accepts an operation of designating the unit distance as the second designated value and an operation of designating the directivity information indicating the characteristics corresponding to the direction of the first wind from the user (S410). In addition, at this time, the second accepting unit 51 may accept an operation indicating the generation position of the first wind, the first wind direction, and the first assumed wind speed from the user. That is, the unit distance, the directivity information, the generation position of the first wind, the first wind direction, and the first assumed wind speed are input by the user operating the second accepting unit 51. This step S410 is equivalent to the second acceptance step.

[0652] Next, the fourth acquisition unit 52 acquires the generation position of the first wind, the first wind direction of the first wind, and the first assumed wind speed, wherein the first assumed wind speed is the wind speed of the first wind at a position that is a unit distance away from the generation position as a reference in the direction of the first wind direction (S420). In addition, the fourth acquisition unit 52 may also acquire directivity information. Here, the fourth acquisition unit 52 acquires the generation position of the first wind, the first wind direction of the first wind, the first assumed wind speed, and the directivity information input in step S410. This step S420 is equivalent to the fourth acquisition step.

[0653] Next, the second generation unit 53 generates fourth target audio information in which the generation position, the first wind direction, the first assumed wind speed, and the directional information are associated with each other ( S430 ). This step S430 corresponds to the second generation step.

[0654] Next, the storage unit 55 stores aerodynamic sound core information including aerodynamic sound data representing wind speed and aerodynamic sound data representing aerodynamic sound generated by the wind representing the wind speed reaching the ear of the listener L (S440). The aerodynamic sound core information may include data representing the distribution of frequency components of the aerodynamic sound. This step S440 corresponds to the storage step.

[0655] The third output unit 54 outputs the fourth target audio information generated by the second generation unit 53 and the aerodynamic sound core information stored in the storage unit 55 to the acoustic signal processing device 400 (S450). This step S450 corresponds to the third output step.

[0656] [Action Example 10]

[0657] Fig.32 1 is a flowchart of Operation Example 10 of the sound signal processing device 400 according to the present modification. Operation Example 10 is an example in which the third processing unit 440 controls the volume of the pneumatic sound.

[0658] like Fig.32 As shown, first, the fifth acquisition unit 410 acquires the fourth target audio information and aerodynamic sound core information output by the information generating device 50, and the second position information indicating the position of the listener L in the virtual space (S510). This step S510 corresponds to the fifth acquisition step.

[0659] Next, the third calculation unit 420 calculates the distance D between the generation position and the listener L based on the generation position included in the fourth object audio information obtained by the fifth acquisition unit 410 and the obtained second position information (S520). In addition, at this time, the third calculation unit 420 calculates the direction between the two points connecting the object (electric fan F) and the listener L based on the generation position included in the fourth object audio information obtained by the fifth acquisition unit 410 and the obtained second position information. This step S520 is equivalent to the third calculation step.

[0660] Next, the third processing unit 440 processes the aerodynamic sound data in such a manner that the volume of the aerodynamic sound is attenuated as the distance D calculated by the third calculating unit 420 increases (S530). More specifically, the third processing unit 440 processes the aerodynamic sound data in such a manner that the volume of the aerodynamic sound is attenuated by the γ-th power of the value obtained by dividing the representative wind speed by the wind speed reaching the ear. This step S530 corresponds to the third processing step.

[0661] The fourth output unit 450 outputs the pneumatic sound data processed by the third processing unit 440 (S540). This step S540 corresponds to the fourth output step.

[0662] [Action Example 11]

[0663] Fig.33 This is a flowchart of Operation Example 11 of the sound signal processing device 400 according to the present modification. Operation Example 11 is an example in which the third processing unit 440 controls the frequency component of the aerodynamic sound.

[0664] like Fig.33 As shown, first, the fifth acquisition unit 410 acquires the fourth target audio information output by the information generating device 50, the aerodynamic sound core information including data indicating the distribution of the frequency components of the aerodynamic sound, and the second position information indicating the position of the listener L in the virtual space (S510f). This step S510f is equivalent to the fifth acquisition step.

[0665] Next, the process of step S520 is performed.

[0666] Next, the third processing unit 440 processes the aerodynamic sound data in such a manner that the distribution of the frequency components of the aerodynamic sound is shifted toward the low frequency side as the distance D calculated by the third calculating unit 420 increases (S530f). More specifically, the third processing unit 440 processes the aerodynamic sound data in such a manner that the distribution of the frequency components of the aerodynamic sound is shifted toward a frequency that is a reciprocal multiple of a value obtained by dividing the representative wind speed by the wind speed reaching the ear (the above-mentioned R1) (S530f). This step S530f corresponds to the third processing step.

[0667] The fourth output unit 450 outputs the pneumatic sound data processed by the third processing unit 440 (S540). This step S540 corresponds to the fourth output step.

[0668] The information generation method related to this variant can generate the fourth object audio information that establishes an association between the generation position, the first wind direction and the first assumed wind speed. The sound signal processing method related to this variant, for example, processes the pneumatic sound data in a manner such that the volume of the pneumatic sound (the second pneumatic sound) attenuates as the distance D between the object and the listener L increases. In addition, for example, the sound signal processing method related to this variant processes the pneumatic sound data in a manner such that the distribution of the frequency component of the pneumatic sound shifts toward the low frequency side as the distance D between the object and the listener L increases. Therefore, the pneumatic sound (the second pneumatic sound) in the virtual space listened to by the listener L becomes a sound similar to the pneumatic sound (the second pneumatic sound) in the real space listened to by the listener L. That is, the listener L will not hear the pneumatic sound (the second pneumatic sound) that makes him feel awkward, and the listener L can get a sense of presence.

[0669] (Variation 7 of Implementation Method)

[0670] Hereinafter, Modification 7 of the embodiment will be described. Hereinafter, the differences from Modification 6 of the embodiment will be mainly described, and the description of the common points will be omitted or simplified.

[0671] [constitute]

[0672] First, the configurations of the information generating device 60 and the sound signal processing device 500 according to Modification 7 of the present embodiment will be described. Fig.34 1 is a block diagram showing the functional configuration of the information generating device 60 and the sound signal processing device 500 according to the present modification.

[0673] The information generating device 60 and the acoustic signal processing device 500 of this modification can suppress the problem of the second aerodynamic sound that makes the listener L feel uncomfortable, as described in (the knowledge that is the basis of this disclosure), as in Modification 6. In this modification, as in Modification 6, the object is the electric fan F.

[0674] In Modification 6, the wind W, i.e., the first wind, which is emitted from the electric fan F as an object in the virtual space is handled. In Modification 7, the first wind and the second wind, which is a wind different from the first wind, are handled. The first wind related to this modification is the same as Modification 6, which is the wind W emitted from the electric fan F as an object in the virtual space. In addition, the second wind does not need to be a wind caused by an object in the virtual space. In this modification, the second wind is a wind generated in nature in the real space and reproduced in the virtual space (hereinafter referred to as natural wind). In addition, since the second wind is a natural wind, it is a wind whose generating position cannot be determined in the virtual space.

[0675] The information generating device 60 generates the fourth object audio information about the first wind and the fifth object audio information about the second wind and outputs them to the sound signal processing device 500. The sound signal processing device 500 acquires the outputted fourth object audio information and fifth object audio information and outputs sound data to the earphone 20 based on the acquired fourth object audio information and fifth object audio information.

[0676] First of all, Fig.34 The information generating device 60 shown will be described.

[0677] The information generating device 60 includes a third receiving unit 61 , a seventh acquiring unit 62 , a fourth generating unit 63 , a sixth output unit 64 , a storage unit 65 , and a display unit 66 .

[0678] The third accepting unit 61 accepts an operation from a user (for example, a creator of content executed in a virtual space) of the information generating device 60. The third accepting unit 61 is specifically implemented by a hardware button, but may be implemented by a touch panel or the like.

[0679] The seventh acquisition unit 62 acquires a generation position of a first wind blowing in the virtual space, a first wind direction and a first assumed wind speed of the first wind, and a second wind direction and a second assumed wind speed of the second wind blowing in the virtual space.

[0680] In this modification, the first wind blowing in the virtual space is the wind W radiated from the electric fan F as the target, as in Modification 6. That is, the location where the first wind is generated is the location where the electric fan F is arranged. In addition, in this modification, as in Modification 6, as long as the first wind blows in the virtual space, the electric fan F as the target radiating the first wind does not need to be arranged in the virtual space (more specifically, the virtual space where the listener L is located). That is, as in Modification 6, the electric fan F as the target radiating the first wind may be arranged in the virtual space where the listener L is located, or may not be arranged in the virtual space where the listener L is located.

[0681] The first wind direction is the direction of the first wind, which is the front direction of the wind W radiated from the object (electric fan F). Fig.19 The first assumed wind speed of this modification example may be any value as long as it indicates the wind speed of the first wind. Here, the first assumed wind speed is the same as in modification example 6, and is the wind speed of the first wind at a position that is a unit distance away from the generating position as a reference in the direction of the first wind direction. That is, the first assumed wind speed is, for example, Fig.19 The wind speed wsF is shown.

[0682] As described above, in the virtual space, since the wind W radiated from the target fan F, that is, the first wind, blows, the listener L hears the aerodynamic sound (second aerodynamic sound) generated by the wind W (first wind) reaching the listener L's ears.

[0683] In addition, the second wind is a natural wind, and the second wind direction is the wind direction of the second wind. For example, when the second wind is a south-southwest wind, the second wind direction indicates south-southwest. In this case, of course, the relationship between the geometric or mathematical direction indicating the direction in the virtual space and the geographical direction indicating the east, west, south, and north needs to be set in advance.

[0684] The second assumed wind speed is the wind speed of the second wind. Since the second wind is a natural wind, the second assumed wind speed has a constant value regardless of the location in the virtual space. That is, the listener L is bathed in the second wind of a constant wind speed regardless of the position of the listener L in the virtual space.

[0685] Furthermore, since the second wind blows in the virtual space, the listener L hears aerodynamic sound (second aerodynamic sound) generated when the second wind reaches the ears of the listener L.

[0686] Therefore, in this modification, the listener L hears at least one of the second aerodynamic sound caused by the first wind and the second aerodynamic sound caused by the second wind.

[0687] In addition, in this modification, the third accepting unit 61 accepts an operation indicating the generation position of the first wind, the first wind direction, the first assumed wind speed, the second wind direction, and the second assumed wind speed from the user. That is, the user operates the third accepting unit 61 to input the generation position of the first wind, the first wind direction, the first assumed wind speed, the second wind direction, and the second assumed wind speed, and the seventh acquisition unit 62 acquires the input generation position of the first wind, the first wind direction, the first assumed wind speed, the second wind direction, and the second assumed wind speed. The details of the process of the third accepting unit 61 accepting the operation indicating the first assumed wind speed and the second assumed wind speed from the user will be described later using the process of the display unit 66.

[0688] The third accepting unit 61 may accept an operation from the user to designate that the unit distance is the second designated value. That is, when the user operates the third accepting unit 61, the unit distance is set to the second designated value.

[0689] The fourth generator 63 generates fourth target audio information associating the generation position, the first wind direction, and the first assumed wind speed acquired by the seventh acquirer 62. The fourth generator 63 generates fifth target audio information associating the second wind direction acquired by the seventh acquirer 62 and the second assumed wind speed.

[0690] In addition, the third receiving unit 61 may receive an operation from the user to designate directional information indicating characteristics corresponding to the direction of the first wind. Fig.19 When such an operation is accepted, the fourth generating unit 63 generates fourth target audio information in which the generation position, the first wind direction, and the first assumed wind speed are associated with the directional information indicated by the operation accepted by the third accepting unit 61 .

[0691] The storage unit 65 is a storage device that stores computer programs and the like executed by the third receiving unit 61, the seventh acquiring unit 62, the fourth generating unit 63, the sixth output unit 64, and the display unit 66. In addition, the storage unit 65 stores aerodynamic sound core information including representative wind speed and aerodynamic sound data, the aerodynamic sound data indicating aerodynamic sound (second aerodynamic sound) generated by the wind representing the wind speed reaching the ears of the listener L in the virtual space.

[0692] In this modification, the third accepting unit 61 accepts an operation indicating the pneumatic sound core information from the user. That is, the user operates the third accepting unit 61 to input the pneumatic sound core information, and the input pneumatic sound core information is stored in the storage unit 65.

[0693] The aerodynamic sound core information shows a representative wind speed indicating an example of a value of one wind speed and aerodynamic sound data indicating aerodynamic sound generated by wind of the representative wind speed reaching the ear of the listener L, and is a database used for information processing in the acoustic signal processing device 500 described later. The aerodynamic sound data indicating aerodynamic sound (second aerodynamic sound) indicates the volume of the second aerodynamic sound, etc.

[0694] The sixth output unit 64 outputs the fourth object audio information generated by the fourth generation unit 63 and the fifth object audio information generated by the fourth generation unit 63. More specifically, the sixth output unit 64 outputs the fourth object audio information generated by the fourth generation unit 63 when the generation position of the first wind is in the virtual space (more specifically, the virtual space where the listener L is located). The sixth output unit 64 outputs the fifth object audio information generated by the fourth generation unit 63 when the generation position of the first wind is not in the virtual space. The sixth output unit 64 outputs the fourth object audio information or the fifth object audio information to the sound signal processing device 500. In addition, the sixth output unit 64 outputs the aerodynamic sound core information stored in the storage unit 65 to the sound signal processing device 500.

[0695] The display unit 66 is a display device that displays an image in which the wind speed is associated with words expressing the wind speed. The display unit 30 is implemented by a display panel such as a liquid crystal panel or an organic EL (Electro Luminescence) panel.

[0696] Fig.35 This is a diagram showing an example of an image displayed on the display unit 66 of this modification. The image is an image in which the wind speed is associated with the words expressing the wind speed. In the image, for example, when the wind speed is 0.0 to 0.2 [m / s], the words expressing the wind speed are "calm". In addition, the description of the wind on "land" and "sea" when the wind blows at the wind speed is displayed. In addition, the image only needs to include characters, graphics, diagrams, etc.

[0697] The user recognizes the image displayed on the display unit 66. Furthermore, the third accepting unit 61 accepts from the user a first operation for designating the wind speed represented by the displayed image as the first assumed wind speed, and a second operation for designating the wind speed represented by the displayed image as the second assumed wind speed. That is, the user operates the third accepting unit 61 to input the first assumed wind speed and the second assumed wind speed, and the seventh acquiring unit 62 acquires the input first assumed wind speed and the second assumed wind speed.

[0698] Furthermore, Fig.34 The audio signal processing device 500 shown will be described.

[0699] The acoustic signal processing device 500 processes the second aerodynamic sound caused by the first wind related to the fourth object audio information when the fourth object audio information is output by the information generating device 60. In addition, the acoustic signal processing device 500 processes the second aerodynamic sound caused by the second wind related to the fifth object audio information when the fifth object audio information is output by the information generating device 60.

[0700] like Fig.34 As shown, the sound signal processing device 500 includes an eighth acquisition unit 510 , a third calculation unit 420 , a fourth processing unit 540 , a seventh output unit 550 , and a storage unit 570 .

[0701] The eighth acquisition unit 510 acquires the fourth object audio information or the fifth object audio information output by the information generating device 60. The eighth acquisition unit 510 acquires the second position information indicating the position of the listener L in the virtual space and the aerodynamic sound core information output by the information generating device 60. The eighth acquisition unit 510 acquires the second position information from the earphone 20 (more specifically, the head sensor unit 21), but is not limited thereto.

[0702] When the eighth acquisition unit 510 acquires the fourth object audio information, that is, when the generation position of the first wind is in the virtual space (more specifically, the virtual space where the listener L is located), the third calculation unit 420 performs the following processing. That is, the third calculation unit 420 calculates the distance D between the generation position (that is, the electric fan F) and the listener L based on the generation position included in the fourth object audio information acquired by the eighth acquisition unit 510 and the acquired second position information. The third calculation unit 420 calculates the distance D by the same method as the first calculation unit 120 of the related embodiment.

[0703] Furthermore, the third calculation unit 420 calculates the direction between two points connecting the object (electric fan F) and the listener L based on the generation position included in the fourth target audio information acquired by the eighth acquisition unit 510 and the acquired second position information.

[0704] When the fourth object audio information is obtained by the eighth acquisition unit 510, the fourth processing unit 540 processes the aerodynamic sound data included in the aerodynamic sound core information based on the position indicated by the second position information obtained by the eighth acquisition unit 510. That is, in this case, the second aerodynamic sound brought by the first wind is processed. More specifically, the fourth processing unit 540, like the third processing unit 440 of the modification example 6, can process the aerodynamic sound data in a manner such that the volume of the aerodynamic sound is attenuated as the distance D calculated by the third calculation unit 420 increases. The distance D calculated by the third calculation unit 420 is a value that depends on the position indicated by the second position information. In this case, the aerodynamic sound data processed by the fourth processing unit 540 is data representing the second aerodynamic sound brought by the first wind.

[0705] When the fifth target audio information is acquired by the eighth acquisition unit 510, the fourth processing unit 540 does not process the aerodynamic sound data included in the aerodynamic sound core information based on the position indicated by the second position information acquired by the eighth acquisition unit 510. Since the second wind is a natural wind, the second assumed wind speed of the second wind represents a constant value regardless of the position of the listener L indicated by the second position information. Therefore, the fourth processing unit 540 processes the aerodynamic sound data regardless of the position indicated by the second position information.

[0706] For example, when the fifth target audio information is acquired, the fourth processing unit 540 performs the following processing.

[0707] If the wind speed of the second wind when it reaches the ear of the listener L, that is, the wind speed reaching the ear, is assumed to be Se2, then Se2 satisfies (Formula 14).

[0708] Se2 = Second assumed wind speed (Formula 14)

[0709] In addition, let the value obtained by dividing the representative wind speed by the wind speed reaching the ear Se2 be R2. Furthermore, the volume V5 of the second aerodynamic sound heard by the listener L satisfies (Formula 15).

[0710] V5 = volume represented by the pneumatic sound data of the pneumatic sound core information × {(1 / R2)}^γ ... (Formula 15)

[0711] The storage unit 570 is a storage device that stores computer programs and the like executed by the eighth acquisition unit 510 , the third calculation unit 420 , the fourth processing unit 540 , and the seventh output unit 550 .

[0712] That is, in this modification, the first wind, which is the wind W radiated from the object, is processed according to the position of the listener L. However, the second wind, which is the natural wind, is not processed according to the position of the listener L, but is processed independently of the position of the listener L.

[0713] Hereinafter, operation example 12 of the information generating method performed by the information generating device 60 and operation example 13 of the sound signal processing method performed by the sound signal processing device 500 will be described.

[0714] [Action Example 12]

[0715] Fig.36 This is a flowchart of Operation Example 12 of the information generating device 60 according to this modification.

[0716] like Fig.36 As shown, first, the display unit 66 displays an image in which the wind speed and the words expressing the wind speed are associated with each other (S610).

[0717] Next, the third accepting unit 61 accepts from the user a first operation of designating the wind speed represented by the displayed image as the first assumed wind speed, and a second operation of designating the wind speed represented by the displayed image as the second assumed wind speed (S620). In addition, at this time, the third accepting unit 61 may accept from the user an operation indicating the generation position of the first wind, the first wind direction, and the second wind direction. That is, by the user operating the third accepting unit 61, the generation position of the first wind, the first wind direction, the first assumed wind speed, the second wind direction, and the second assumed wind speed are input. This step S620 is equivalent to the third acceptance step. ...

Claims

1. A method for processing an acoustic signal, wherein: include: a first acquisition step of acquiring object information and second position information, wherein the object information includes first position information indicating a position of an object in a virtual space, first sound data indicating a first sound caused by the object, and first identification information indicating a means for processing the first sound data, and the second position information indicating a position of a listener of the first sound in the virtual space; a first calculation step of calculating the distance between the object and the listener based on the first position information included in the acquired object information and the acquired second position information; a step of determining, based on the first identification information included in the acquired object information, a processing means for processing the first sound data between a first processing means and a second processing means, wherein the first processing means processes the volume corresponding to the calculated distance, and the second processing means processes the volume corresponding to the calculated distance in a form different from that of the first processing means; A first processing step of processing the first sound data by the determined processing means; as well as The first output step is to output the processed first audio data.

2. The sound signal processing method according to claim 1, wherein: The first processing means is a processing means for processing the first sound data in such a manner that the volume is attenuated inversely proportional to an increase in the calculated distance. The second processing means is a processing means for processing the first sound data in such a manner that the volume increases or decreases in a manner different from that of the first processing means as the calculated distance increases.

3. The sound signal processing method according to claim 2, wherein: The obtained object information includes: second sound data representing a second sound caused by the object, the second sound being different from the first sound; and The second identification information indicates a processing method for the second sound data. In the determination step, a processing means for processing the second sound data is determined between the first processing means and the second processing means based on the second identification information included in the acquired object information. In the first processing step, the second sound data is processed by the determined processing means. In the first output step, the processed second sound data is output. The object is an object associated with a plurality of audio data including the first audio data and the second audio data.

4. The sound signal processing method according to claim 2, wherein: The second processing means is a processing means for processing the first sound data in such a manner that the volume is attenuated by the xth power of the distance, where x≠1.

5. The sound signal processing method according to claim 4, wherein: The first identification information indicates that the processing means for the first audio data is the second processing means, and a value of x.

6. The sound signal processing method according to claim 4, wherein: In the case where the first sound is a pneumatic sound generated as the object moves, The first identification information indicates that the processing means for the first audio data is the second processing means, and that x is α, α is a real number, and α satisfies the following formula: α>1。 7. The sound signal processing method according to claim 6, wherein: In a case where the first sound is aerodynamic sound generated by wind radiated from the object and reaching the ears of the listener, The first identification information indicates that the processing means for the first audio data is the second processing means, and that x is β, β is a real number, and β satisfies the following formula: β>2。 8. The sound signal processing method according to claim 7, wherein: The above α and the above β satisfy the following formula: α<β。 9. The sound signal processing method according to claim 7, wherein: The method includes a first receiving step of receiving an operation of designating a value of the α or β from a user.

10. The sound signal processing method according to claim 2, wherein: The first identification information is information indicating whether the first processing means is executed. In the above decision step, Based on the obtained first identification information, it is determined whether to execute the first processing means, and, Regardless of whether the first processing means is executed, it is determined to execute the second processing means. The second processing means is a processing means for processing the first sound data so that the volume becomes a predetermined value when the calculated distance is within a predetermined threshold value.

11. The sound signal processing method according to claim 10, wherein: The threshold value specified above is a value corresponding to the personal space.

12. The sound signal processing method according to claim 10, wherein: The method includes a first accepting step of accepting an operation from a user to designate that the predetermined threshold value is a first designated value.

13. A method for generating information, wherein: include: a second acquisition step of acquiring first sound data representing a first sound generated at a position associated with the position of the listener in the virtual space, and first position information representing a position of an object in the virtual space; as well as The first generation step generates the first object audio information based on the first sound data obtained and the first position information obtained, and the first object audio information includes information related to the object and the first position information that reproduces the first sound at a position associated with the position of the listener due to the object.

14. The information generating method according to claim 13, wherein: The above objects radiate wind, The said listener bathes in the said wind which is radiated, The first sound is aerodynamic sound generated by the wind radiated from the object reaching the ears of the listener.

15. The information generating method according to claim 14, wherein: In the first generation step, the first object audio information further including the unit distance information is generated. The unit distance information is information including a unit distance serving as a reference distance and pneumatic sound data indicating the pneumatic sound at a position away from the position of the object by the unit distance.

16. The information generating method according to claim 15, wherein: In the first generation step, the first object audio information further including the directivity information is generated. The directional information is information indicating characteristics corresponding to the direction of the radiated wind. The aerodynamic sound data indicated by the unit distance information is data indicating the aerodynamic sound at a position away from the position of the object by the unit distance in a front direction in which the wind is radiated by the object indicated by the directional information.

17. The information generating method according to claim 13, wherein: In the above-mentioned first generation step, the above-mentioned first object audio information is generated, which also includes flag information, and the above-mentioned flag information indicates whether to perform convolution processing of the first sound signal based on the above-mentioned first sound data depending on the head-related transfer function processing of the direction of arrival of the sound when reproducing the above-mentioned first sound, and the above-mentioned first sound data represents the above-mentioned first sound generated from the above-mentioned object.

18. A method for processing an audio signal, wherein: include: a third acquisition step of acquiring the first target audio information generated by the information generation method according to claim 13, the acquired first sound data, and second position information indicating the position of the listener of the first sound; a second calculation step of calculating the distance between the object and the listener based on the first position information included in the acquired first object audio information and the acquired second position information; a second processing step of processing the first sound data in such a manner that the volume of the first sound is attenuated as the calculated distance increases; as well as The second output step is to output the processed first audio data.

19. A method for processing an audio signal, wherein: include: a third acquisition step of acquiring the first target audio information generated by the information generation method according to claim 14, the acquired first sound data, and second position information indicating the position of the listener of the first sound; a second calculation step of calculating the distance between the object radiating the wind and the listener based on the first position information included in the acquired first object audio information and the acquired second position information; a second processing step of processing the first sound data in such a manner that the volume of the first sound is attenuated as the calculated distance increases; as well as The second output step is to output the processed first audio data.

20. A method for processing an audio signal, wherein: include: a third acquisition step of acquiring the first target audio information generated by the information generation method according to claim 15, the acquired first sound data, and second position information indicating the position of the listener of the first sound; a second calculation step of calculating the distance between the object radiating the wind and the listener based on the first position information included in the acquired first object audio information and the acquired second position information; a second processing step of processing the first sound data in such a manner that, when the calculated distance is greater than the unit distance indicated by the unit distance information included in the acquired first object audio information, the volume of the first sound is attenuated according to the calculated distance and the unit distance; as well as The second output step is to output the processed first audio data.

21. A method for processing an audio signal, wherein: include: a third acquisition step of acquiring the first target audio information generated by the information generation method according to claim 16, the acquired first sound data, and second position information indicating the position of the listener of the first sound, A second calculation step of calculating the distance between the object radiating the wind and the listener and the direction between two points connecting the object and the listener based on the first position information included in the acquired first object audio information and the acquired second position information; A second processing step of processing the first sound data in a manner that controls the volume of the first sound based on the angle between the front direction and the calculated direction between the two points and the characteristic represented by the directivity information, and in a manner that, when the calculated distance is greater than the unit distance represented by the unit distance information included in the acquired first object audio information, the volume of the first sound is attenuated according to the calculated distance and the unit distance; as well as The second output step is to output the processed first audio data.

22. A method for processing an audio signal, wherein: include: A third acquisition step, acquiring the first object audio information generated by the information generation method according to any one of claims 13 to 16, the first sound data obtained, and the second object audio information in which the first position information is associated with the second sound data representing the second sound caused by the object; a second processing step of not convolving a first sound signal based on the first sound data obtained with a head-related transfer function that depends on the direction of arrival of the sound, and convolving a second sound signal based on the second sound data represented by the second object audio information obtained with a head-related transfer function that depends on the direction of arrival of the sound; as well as The second output step is to output the unprocessed first audio signal and the processed second audio signal.

23. A method for processing an audio signal, wherein: include: A third acquisition step, acquiring the first object audio information generated by the information generation method according to any one of claims 14 to 16, the first sound data obtained, and the second object audio information in which the first position information is associated with the second sound data representing the second sound caused by the object; A second processing step of performing a process dependent on the direction of wind arrival on the first sound signal based on the first sound data obtained, and performing a process dependent on the direction of sound arrival convolving a head-related transfer function on the second sound signal based on the second sound data represented by the second object audio information obtained; as well as The second output step is to output the processed first audio signal and the processed second audio signal.

24. A method for processing an audio signal, wherein: include: a third acquisition step of acquiring the first object audio information generated by the information generation method according to any one of claims 14 to 16, the acquired first sound data, and third object audio information, wherein the third object audio information associates third position information indicating the position of other objects in the virtual space with third sound data indicating the third sound generated at the position of the other objects, the other objects being objects different from the above-mentioned object; A second processing step of performing a process dependent on the direction of wind arrival on the first sound signal based on the first sound data obtained, and performing a process dependent on the direction of sound arrival convolving a head-related transfer function on the third sound signal based on the third sound data represented by the third object audio information obtained; as well as The second output step is to output the processed first audio signal and the processed third audio signal.

25. A method for generating information, wherein: include: a fourth acquisition step of acquiring a generation position of a first wind blowing in the virtual space, a first wind direction of the first wind, and a first assumed wind speed as the wind speed of the first wind; a second generation step of generating fourth target audio information by associating the obtained generation position, the first wind direction, and the first assumed wind speed; a storing step of storing aerodynamic sound core information including a representative wind speed and aerodynamic sound data, wherein the aerodynamic sound data represents aerodynamic sound generated by the wind representing the wind speed reaching the ears of the listener in the virtual space; and The third output step outputs the generated fourth object audio information and the stored aerodynamic sound core information.

26. The information generating method according to claim 25, wherein: The first assumed wind speed is a wind speed of the first wind at a position that is away from the generation position by a unit distance that serves as a reference distance in the direction of the first wind direction.

27. The information generating method according to claim 26, wherein: The method includes a second accepting step of accepting an operation from a user to designate that the unit distance is a second designated value.

28. The information generating method according to claim 26, wherein: The method comprises a second receiving step of receiving an operation of designating directional information from a user, wherein the directional information indicates a characteristic corresponding to the direction of the first wind. In the second generation step, the fourth target audio information is generated by associating the acquired generation position, the first wind direction, and the first assumed wind speed with the directional information indicated by the accepted operation.

29. A method for processing an audio signal, wherein: include: a fifth acquisition step of acquiring the fourth target audio information and the aerodynamic sound core information output by the information generation method according to any one of claims 26 to 28, and second position information indicating the position of the listener in the virtual space; a third calculation step of calculating the distance between the generation position and the listener based on the generation position included in the fourth object audio information and the second position information; A third processing step of processing the pneumatic sound data in such a manner that the volume of the pneumatic sound decreases as the calculated distance increases; and The fourth output step is to output the processed pneumatic sound data.

30. The sound signal processing method according to claim 29, wherein: In the third processing step, the aerodynamic sound data is processed based on the wind speed of the first wind when it reaches the ear of the listener, that is, the wind speed reaching the ear. Regarding the wind speed reaching the ear, the wind speed decreases as the calculated distance increases.

31. The sound signal processing method according to claim 30, wherein: The wind speed reaching the ear is a value attenuated by the z-th power of a value obtained by dividing the calculated distance by the unit distance.

32. The sound signal processing method according to claim 31, wherein: The above z satisfies the following formula: z＝1。 33. The sound signal processing method according to claim 31, wherein: In the third processing step, the aerodynamic sound data is processed so that the volume of the aerodynamic sound is attenuated by a value raised to the γth power of a value obtained by dividing the representative wind speed by the wind speed reaching the ear.

34. The sound signal processing method according to claim 33, wherein: The above γ and the above z satisfy the following formula: γ×z<4.

35. A method for processing an audio signal, wherein: include: a fifth acquisition step of acquiring the fourth target audio information output by the information generation method according to any one of claims 26 to 28, the aerodynamic sound core information including data indicating the distribution of the frequency components of the aerodynamic sound, and the second position information indicating the position of the listener in the virtual space; a third calculation step of calculating the distance between the generation position and the listener based on the generation position included in the fourth object audio information and the second position information obtained; A third processing step of processing the aerodynamic sound data in such a manner that the distribution of the frequency component of the aerodynamic sound is shifted toward a lower frequency side as the calculated distance increases; and The fourth output step is to output the processed pneumatic sound data.

36. The sound signal processing method according to claim 35, wherein: In the third processing step, the aerodynamic sound data is processed based on the wind speed of the first wind when it reaches the ear of the listener, that is, the wind speed reaching the ear. Regarding the wind speed reaching the ear, the wind speed decreases as the calculated distance increases.

37. The sound signal processing method according to claim 36, wherein: The wind speed reaching the ear is a value attenuated by the z-th power of a value obtained by dividing the calculated distance by the unit distance.

38. The sound signal processing method according to claim 37, wherein: The above z satisfies the following formula: z＝1。 39. The sound signal processing method according to claim 36, wherein: In the third processing step, the aerodynamic sound data is processed so that the distribution of the frequency component of the aerodynamic sound is shifted to a frequency that is a reciprocal multiple of a value obtained by dividing the representative wind speed by the wind speed reaching the ear.

40. A method for generating information, wherein: include: a sixth acquisition step of acquiring a second wind direction of a second wind blowing in the virtual space and a second assumed wind speed which is the wind speed of the second wind; a third generation step of generating fifth target audio information by associating the obtained second wind direction with the second assumed wind speed; a storing step of storing aerodynamic sound core information including a representative wind speed and aerodynamic sound data, wherein the aerodynamic sound data represents aerodynamic sound generated by the wind representing the wind speed reaching the ears of the listener in the virtual space; and The fifth output step outputs the generated fifth object audio information and the stored aerodynamic sound core information.

41. A method for generating information, wherein: include: a seventh acquisition step of acquiring a generation position of a first wind blowing in the virtual space, a first wind direction of the first wind, a first assumed wind speed as the wind speed of the first wind, a second wind direction of a second wind blowing in the virtual space, and a second assumed wind speed as the wind speed of the second wind; a fourth generation step of generating fourth object audio information that associates the obtained generation position, the first wind direction, and the first assumed wind speed, and generating fifth object audio information that associates the obtained second wind direction with the second assumed wind speed; and The sixth output step is to output the generated fourth object audio information and the generated fifth object audio information.

42. The information generating method according to claim 41, wherein: In the sixth output step, when the generation position of the first wind is in the virtual space, the generated fourth object audio information is output.

43. The information generating method according to claim 42, wherein: In the sixth output step, when the generation position of the first wind is not in the virtual space, the generated fifth object audio information is output.

44. The information generating method according to claim 41, wherein: The method comprises a storage step of storing aerodynamic sound core information including a wind speed representative and aerodynamic sound data, wherein the aerodynamic sound data represents aerodynamic sound generated by the wind representing the wind speed reaching the ears of the listener in the virtual space, In the sixth output step, the stored aerodynamic sound core information is output.

45. The information generating method according to claim 44, wherein: include: A display step of displaying an image in which the wind speed is associated with a word expressing the wind speed; as well as The third accepting step accepts a first operation of designating the wind speed represented by the displayed image as the first assumed wind speed, and accepts a second operation of designating the wind speed represented by the displayed image as the second assumed wind speed.

46. ​​A method for processing an audio signal, wherein: include: an eighth acquisition step of acquiring second position information indicating the position of the listener in the virtual space, and the fourth object audio information or the fifth object audio information output by the information generation method according to claim 44; a fourth processing step of processing the aerodynamic sound data included in the aerodynamic sound core information based on the position indicated by the second position information obtained when the fourth target audio information is obtained, and not processing the aerodynamic sound data included in the aerodynamic sound core information based on the position indicated by the second position information obtained when the fifth target audio information is obtained; and The seventh output step is to output the processed pneumatic sound data.

47. A computer program for causing a computer to execute the sound signal processing method according to any one of claims 1 to 12, 18 to 21 and 46.

48. A computer program for causing a computer to execute the information generating method according to any one of claims 13 to 17, 25 to 28, and 40 to 45.

49. An audio signal processing device, wherein: have: a first acquisition unit that acquires object information and second position information, the object information including first position information indicating a position of an object in a virtual space, first sound data indicating a first sound caused by the object, and first identification information indicating a means for processing the first sound data, the second position information indicating a position of a listener of the first sound in the virtual space; a first calculation unit that calculates a distance between the object and the listener based on the first position information included in the acquired object information and the acquired second position information; a determination unit that determines, based on the first identification information included in the acquired object information, a processing means for processing the first sound data between a first processing means and a second processing means, the first processing means performing a volume processing corresponding to the calculated distance, and the second processing means performing a volume processing corresponding to the calculated distance in a form different from that of the first processing means; a first processing unit for processing the first sound data by using the determined processing means; as well as The first output unit outputs the processed first audio data.

Images